acceptodds
Under review as a conference paper at ICLR 2027

HarnessIR: An Ontology-Guided Intermediate Representation for Cross-Runtime Transfer of Agent Harnesses

Abstract

Agent Harnesses encode reusable operational knowledge that determines how an agent organizes and executes under different runtime conditions. However, this knowledge is often tightly coupled with framework-specific control structures and runtime-specific execution designs, making it difficult to reuse and verify a tuned agent system once it is separated from its original implementation. We study how to represent harness-level behavioral knowledge independently of a specific runtime and preserve its key behavioral constraints when the implementation environment changes. To address this problem, we introduce HarnessIR, an ontology-based, runtime-independent intermediate representation (IR) for representing and reusing behavioral designs embedded in Agent Harnesses. A Harness ontology provides a common conceptual foundation for Harness execution, allowing framework-specific constructs from different runtimes to be related within a shared semantic space. Based on this foundation, HarnessIR separates a Harness into semantic mechanisms and execution policies: the former capture reusable semantic objects and their stable dependencies, while the latter specify how these mechanisms are selected, composed, and constrained under different execution conditions. Through this intermediate representation, behavioral designs accumulated in a Harness can be separated from a specific framework implementation and realized again in heterogeneous runtimes. We evaluate HarnessIR on HarnessIR-Bench, a benchmark we construct for systematic evaluation. Across 10 source Harnesses, HarnessIR achieves 45.0% Behavior Success, compared with 34.0% for a static ontology-based baseline, indicating improved reuse and preservation of Harness behavior across heterogeneous implementations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Harness, Ontology, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FMv9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates cross-runtime transfer of agent harnesses, emphasizing that producing a plausible final answer is not equivalent to preserving the source harness's execution behavior. The proposed HarnessIR uses a fixed harness ontology to represent states, actions, decisions, constraints, and related entities. The method combines ontology-guided static semantic extraction from AST-partitioned source programs with policy inference from source-native execution traces. The recovered semantic graph specifies state- and decision-dependent action rules with associated constraints. A separate target-runtime semantic map, obtained from target framework documentation, guides the realization of the graph as a native target harness.

The authors also introduce HarnessIR-Bench, comprising 18 source-harness entries, with six held-out evaluation scenarios per test source and transfers to LangGraph and Microsoft Agent Framework. Across deepseek-v4-flash and GLM-5.2, the reported mean Behavior Success is 45.00%, versus 34.03% for the strongest static ontology-based baseline, SOT. The paper also reports task-level and joint end-to-end metrics, dimension-level analyses, and component ablations.

Critical strengths

1. Practical and well-motivated research problem.

The paper addresses a practical and increasingly relevant problem: transferring agent harnesses across heterogeneous runtimes while preserving their original behavior. As agent frameworks continue to evolve, the ability to reuse existing harnesses without losing important behavioral constraints is valuable and well motivated.

2. An appropriate application of the compiler IR paradigm.

The paper draws a natural and compelling analogy to intermediate representations in compiler-based code translation. Introducing an ontology-guided IR to decouple source harness semantics from target-runtime implementations is a well-motivated design choice that provides a clear and systematic approach to cross-runtime harness transfer.

Critical weaknesses

While the paper evaluates behavioral preservation after cross-runtime transfer, it does not directly assess the accuracy and completeness of the recovered HarnessIR. In particular, it remains unclear whether the extracted execution policies faithfully capture the source harness's guards, state dependencies, and behavioral constraints, especially for branches not observed in the available execution traces.

Questions for authors

Why is the evaluation limited to general-purpose agent frameworks? Have the authors considered evaluating HarnessIR on widely used coding-agent systems such as Codex and Claude Code? Such an evaluation could provide stronger evidence of the practical applicability and generalizability of HarnessIR.

3 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1

It is encouraging that you recognize the significance of our research and the motivation behind our IR-based approach. We highly value your thoughtful feedback, which helps us further improve this work. We respond to each point below.

For W1: Semantic recovery, traceability, and IR evaluation limitations.

First, we would like to clarify that HarnessIR does not recover execution policies solely from execution traces. Instead, it combines static semantic analysis of the source code with dynamic execution evidence, leveraging their complementary information to recover the Harness's semantic structure and execution policies.

For branches not observed in the execution traces, static code analysis provides a basis for recovering the explicit control structures defined in the source code, without relying entirely on dynamic observations. Meanwhile, during the recovery process, we preserve the corresponding source code snippets and execution traces as evidence for all recovered semantic entities, relations, and execution rules, enabling their traceability and verification with respect to the recovered Guard Conditions, State Dependencies, and Behavioral Constraints. When the evidence is insufficient or ambiguous, we preserve the uncertainty rather than infer unverified runtime behavior. This mechanism aims to mitigate information loss caused by limited execution traces and avoid generating unsupported execution policies.

We also acknowledge that our current experiments primarily validate HarnessIR indirectly through behavioral preservation after migration, without independently evaluating the accuracy and completeness of the recovered IR. Therefore, directly evaluating IR-level recovery quality, particularly the recovery of unobserved branches, remains a limitation of this work and an important direction for future research.

For Q1: Coding agent evaluation.

We initially selected general-purpose agent frameworks such as LangGraph and Microsoft Agent Framework because they provide explicit programming interfaces that facilitate the direct implementation of target Harness components, including state management, control flow, and action execution logic. This choice was primarily motivated by the experimental setup and the implementation interfaces provided by the target runtimes, rather than a limitation of HarnessIR to general-purpose agent frameworks.

To further evaluate the applicability of HarnessIR to specialized coding-agent systems, we followed your suggestion and added Codex and Claude Code as target systems in supplementary experiments. It is worth noting that although both Codex and Claude Code provide extension mechanisms for external integration, unlike LangGraph, they do not directly expose programming interfaces for constructing an entire Harness. Therefore, in our experiments, we use their respective tool interfaces, MCP, and Hooks to accommodate and implement the migrated execution policies, rather than directly controlling or replacing their entire internal execution mechanisms. The results are as follows:

Target HarnessBehaviorStateTaskTask-JE2EE2E-J
Claude Code32.22%50.64%14.44%28.89%11.11%14.44%
Codex37.22%55.77%14.44%30.56%12.22%15.56%

The results demonstrate that HarnessIR can transfer source Harness behaviors to these two specialized coding-agent systems, achieving non-trivial behavioral preservation and end-to-end task execution. Meanwhile, because the native execution mechanisms and extension interfaces of these systems differ from those of general-purpose agent frameworks, some source-Harness state dependencies and execution constraints may be difficult to fully realize through the available extension mechanisms, which may also affect the resulting behavioral preservation. Nevertheless, these supplementary experiments extend the target-runtime coverage of HarnessIR beyond general-purpose agent frameworks and provide additional empirical evidence of its practical applicability and generalizability across different types of agent systems. We have added the coding-agent transfer experiments to Section 5.6 of the revised manuscript.

We sincerely appreciate your time and thoughtful consideration of our responses. We hope our clarifications have addressed your concerns, and would be grateful for any further questions or suggestions you may have.

Reviewer_FMv9OpenReview ↗

Reply to above section

Thank you for the detailed response and for conducting additional experiments on Codex and Claude Code. I appreciate the clarification regarding the complementary roles of static analysis and execution traces in semantic recovery. The additional experiments strengthen the practical relevance of the work and help address my concerns about its broader applicability. However, the accuracy and completeness of the recovered IR, particularly for branches not observed in the available execution traces, remain insufficiently validated. Therefore, I have decided to maintain my original rating (Weak Accept).

AuthorsOpenReview ↗

Response to the Follow-up Comments

Thank you very much for your thoughtful feedback and positive assessment of our work.

We acknowledge that our response may not have fully addressed your concern regarding the accuracy and completeness of IR recovery, particularly for branches not covered by the available execution traces. We sincerely appreciate this observation and will continue to refine the semantic recovery process and explore more effective approaches to validating the recovery of unobserved branches in our future work.

Thank you again for your time and constructive suggestions.

Reviewer_VnqdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses an important problem of how to represent harness level behavioral knowledge independently of a specific runtime and preserve its behavioral constraints when the implementation runtime changes. The authors introduce HarnessIR, an ontology-based, runtime-independent intermediate representation (IR) for representing and reusing behavioral designs. HarnessIR separates a Harness into semantic mechanisms and execution policies. The authors also propose HarnessIR-Bench for systematic evaluation.

Critical strengths
  • The paper identifies an important problem of cross runtime transfer and aims to address this problem through a novel method of HarnessIR. This method seems to be intuitive and easy to understand, while the authors also show that this method is effective using HarnessIR-Bench.
  • In addition to the method HarnessIR that the authors proposed, the authors also introduced a new benchmark HarnessIR-Bench to evaluate semantic and behavioral preservation in cross-runtime harness transfer.
Critical weaknesses
  • Although the authors explained why they introduced ontology guided IR, but didn't explain intuitively and empirically why this could work.
  • It seems that in the E2E setting, there is no clear statistically significant gain from using HarnessIR compared to using other baseline methods. Therefore, it is hard to tell whether HarnessIR is actually useful.
Questions for authors
  • Could the authors show an actual example of the IR?
5 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Weakness 2

We are grateful for your positive assessment of HarnessIR and HarnessIR-Bench. The insightful suggestions provide valuable guidance for further strengthening our work. Our detailed responses are presented below.

For W1: Rationale and ablation evidence for ontology guidance.

The effectiveness of ontology-guided IR primarily stems from the unified semantic foundation it provides. Although different Agent Runtimes adopt different programming interfaces and execution constructs, their underlying Harness behaviors share common semantic elements, such as State, Action, Decision, and Constraint. The complete ontology, including its entity types, relations, and subtypes, is detailed in Appendix B. The ontology provides a unified semantic space through predefined entity types, attributes, and relations, allowing framework-specific constructs to be abstracted into a common representation with explicit semantics and dependencies. Importantly, this representation captures not only the semantic entities within a Harness but also the dependencies among State, Decision, Action, and Constraint, thereby helping recover and preserve the state conditions, decision conditions, and execution constraints governing action execution. The same semantic representation can then be aligned with target-native constructs through the ontology, reducing semantic mismatches during cross-runtime transfer.

Furthermore, the ablation study in Table 2 provides empirical evidence for the contribution of Ontology Guidance. Removing Ontology Guidance reduces Behavior from 49.44% to 36.94% and from 40.56% to 35.00% under DeepSeek-V4-Flash and GLM-5.2, respectively. E2E also decreases from 11.11% to 1.67% and from 13.61% to 5.00%, respectively. The consistent performance drops across both models indicate that ontology-guided semantic alignment helps recover the source Harness's semantic structure and execution dependencies, thereby improving behavioral preservation across runtimes.

For W2: E2E interpretation, statistical significance, and joint task–behavior analysis.

Regarding the statistical significance of the E2E improvements, we first clarify the evaluation criteria of this metric. E2E is a strict joint metric, requiring an execution to simultaneously satisfy final task success (Task), behavioral preservation (Behavior), and successful execution in the target runtime. Therefore, even if a migrated Harness preserves the source Harness's execution behavior, it is still counted as an E2E failure if the final task is not completed. Given that the primary objective of HarnessIR is behavioral preservation across runtimes, we believe its effectiveness should be evaluated by considering both Behavior and E2E.

In the main experiments, HarnessIR achieves the highest E2E rates under both DeepSeek-V4-Flash and GLM-5.2 (11.11% and 13.61%, respectively), while demonstrating more substantial advantages in Behavior (49.44% and 40.56%, respectively).

To further address your concern regarding statistical significance, we conducted five independent reconstruction rounds for both HarnessIR and the strongest baseline, SOT, repeating their complete construction pipelines in each round. Based on the results, we calculated the means and standard deviations across the five rounds and performed two-sided exact permutation tests. The results show that HarnessIR achieves statistically significant improvements over SOT in both E2E (11.17% vs. 9.83%, p = 0.0159) and E2E-J (22.50% vs. 19.06%, p = 0.0079). The complete experimental setup and results are provided in the newly added Appendix A.

Finally, the joint Task-Behavior analysis in Appendix F further shows that, compared with the strongest baseline SOT, HarnessIR increases the proportion of executions that successfully satisfy both Task and Behavior (12.4% vs. 10.7%) and reduces the proportion of executions that fail on both criteria (48.5% vs. 57.5%). These results demonstrate that the value of HarnessIR lies not only in improving final task success, but also in reducing behavioral deviations during cross-runtime transfer, thereby increasing the likelihood of achieving behavioral preservation and task completion simultaneously.

AuthorsOpenReview ↗

Response to Question 1

For Q1: A concrete HarnessIR example and target runtime mapping.

To illustrate the concrete representation of HarnessIR, we use an order-refund Agent as an example. Suppose the source Harness contains three primary operations: 〈Lookup Order〉, 〈Request Human Approval〉, and 〈Execute Refund〉. It specifies that order lookup can be retried at most twice and that refunds exceeding 100 require human approval.

HarnessIR represents the behavioral knowledge of this Harness at two levels.

(1) Semantic Mechanisms.

HarnessIR first uses the predefined ontology to abstract the semantic objects in the source Harness into unified typed entities:

  • State: 〈Order Information〉, 〈Lookup Count〉, and 〈Approval Status〉, representing the states maintained during execution.

  • Action: 〈Lookup Order〉, 〈Request Human Approval〉, 〈Execute Refund〉, and 〈Terminate Workflow〉, representing the operations that the Harness can perform.

  • Decision: 〈Refund Eligibility Check〉 and 〈Approval Requirement Check〉, representing conditional decisions during execution.

  • Constraint: 〈Refund Execution Constraint〉 and 〈Lookup Retry Constraint〉, specifying the refund eligibility and approval requirements, as well as the maximum number of order lookup retries, respectively.

Furthermore, HarnessIR explicitly captures the semantic dependencies among these entities. For example, the 〈Order Information〉 produced by 〈Lookup Order〉 provides the basis for subsequent refund eligibility checks, while 〈Approval Status〉 serves as a prerequisite for executing 〈Execute Refund〉.

(2) Execution Policies.

Based on the Semantic Mechanisms, HarnessIR further describes action selection and its associated constraints under different execution conditions through condition-driven rules. For example:

  • When 〈Order Lookup Failed〉 and 〈Retries Remaining〉 hold, 〈Lookup Order〉 is selected again, with at most two retries allowed. When 〈Retry Limit Reached〉 holds, 〈Terminate Workflow〉 is selected.

  • When 〈Refund Eligibility Not Satisfied〉 holds, 〈Terminate Workflow〉 is selected, prohibiting refund execution.

  • When refund eligibility is satisfied and 〈Refund Amount at Most 100〉 holds, 〈Execute Refund〉 is selected.

  • When refund eligibility is satisfied and 〈Refund Amount Exceeds 100〉 holds, 〈Request Human Approval〉 is selected. 〈Execute Refund〉 is permitted only after 〈Human Approval Granted〉. If approval is denied, the workflow is terminated.

For example, when an order is eligible for a refund of 200 but human approval is denied, the execution policy requires the workflow to terminate. Although 〈Execute Refund〉 is an available action in the Harness, it cannot be executed by bypassing the approval constraint.

Thus, Semantic Mechanisms describe reusable semantic entities and their dependencies, whereas Execution Policies specify how these entities are selected, composed, and constrained under different execution conditions. Together, they constitute the intermediate representation of Harness-level behavioral knowledge.

During target-runtime realization, these runtime-independent semantic entities and execution policies are further mapped to target-native constructs. For example, State can be mapped to state data, Action to execution nodes, and Decision and Constraint to conditional routing and execution logic in the target framework.

We hope our responses have adequately addressed your concerns. We greatly appreciate your patience and time in reviewing our work, and would be grateful for any further questions or suggestions.

Dear Reviewer, we would like to kindly follow up on our response to your review. Have our explanations helped resolve the questions you raised? If any concerns remain, or if new questions have come up, we would welcome the opportunity to address them. Thank you for your time and consideration.

Reviewer_kvBbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles behavior-preserving migration of agent harnesses from one framework (Mastra, Agno, CAMEL, DSPy and others) to another runtime (LangGraph or Microsoft Agent Framework). HarnessIR uses a fixed ontology (State, Action, Decision, Constraint and a few other types) as an intermediate representation. An LLM maps the source code and execution traces into this representation, maps each type to constructs in the target framework, and then generates the target code.

The authors also introduce HarnessIR-Bench, covering 10 test harnesses, 2 target runtimes and 360 executions per method and backbone. HarnessIR achieves 49.44% and 40.56% Behavior success with the two backbones, compared with 33.33% and 34.72% for the strongest baseline. However, Task success remains around 19% for both.

Overall, the paper addresses a practical and worthwhile problem in agent development with a structured approach to framework migration. The improvements in behavior preservation are promising, although the relatively low end-to-end task success suggests that reliable migration remains challenging.

Critical strengths

I identified two main strengths that I particularly value in this paper:

1. A practical problem with clear motivation. Migrating harnesses across agent frameworks is a real challenge, and the paper highlights an important distinction: a migrated harness may run without errors but still behave differently from the original, as illustrated by the revise-and-approve example in Fig. 1(b). I appreciate that the authors recognize this issue and evaluate behavioral consistency separately from final task outcomes.

2. Ablation results that support the core design choices. The ablations provide useful evidence for the importance of the proposed components. Removing target-native mapping roughly halves Behavior success (from 49.44% to 23.89% with DeepSeek), while removing ontology guidance reduces end-to-end success to 1.67% and 5.00% (Tab. 2). These results help justify the design choices rather than relying only on the overall performance gains.

Critical weaknesses

I also have two major concerns that affect my assessment of the paper:

1. The benchmark is limited, and the behavioral scoring is not sufficiently explained. The evaluation covers only ten single-task workflows, with each target program generated just once, so the variance from LLM-based construction is not measured. At the source level, HarnessIR outperforms the strongest baseline on 4 sources and underperforms on 1 with DeepSeek (6 and 3 with GLM). With such a small test set, it is hard to tell how robust these gains are. My bigger concern is the evaluation protocol. The paper does not clearly explain how source executions are turned into behavioral obligations or what exactly the evaluator checks in target runs. The five behavior dimensions also closely follow the proposed ontology. If HarnessIR produces ontology-aligned states and actions that are easier for the evaluator to recognize than those of the baselines, the comparison may not be entirely fair. The exact-zero results for some baselines, particularly Direct Code Translation's 0% Task and end-to-end success under both backbones, make this concern worth investigating.

2. The end-to-end improvements are modest, and the IR itself is not sufficiently defined. Task success is nearly identical to the strongest baseline (18.61% vs. 18.33% and 19.17% vs. 20.00%), while end-to-end success improves by only 4 and 8 out of 360 executions. This also means that roughly 86%–89% of HarnessIR transfers still fail. On the methodology side, the approach appears to be a sequence of mostly LLM-driven steps rather than a clearly specified intermediate representation. The paper provides no explicit output schemas, execution semantics for the IR, or systematic verification that the generated program preserves the source behavior beyond runtime validation of LLM decisions. I therefore find the comparisons with LLVM, MLIR, and translation validation somewhat stronger than what the current implementation supports.

Overall, these concerns mainly relate to the reliability of the evaluation and the technical grounding of the proposed method. If the authors can demonstrate that the evaluator is method-neutral and that the improvements hold across repeated constructions, I would be open to reconsidering my score.

Questions for authors
  1. How exactly is behavioral success evaluated? Could the authors clarify how behavioral obligations are derived from source executions, what the evaluator observes in target runs, whether an LLM is involved, and whether all methods are instrumented in the same way? If the evaluation differs across methods, could the authors provide results using a method-neutral observer? I would also appreciate an explanation for why Direct Code Translation and SSIR achieve exactly zero success.

  2. How stable are the results across repeated constructions? Could the authors regenerate each target program multiple times, including the recovery, grounding, and code-generation stages, and report per-source results with a source-level significance test? This would help establish whether the observed improvements are consistent rather than dependent on individual LLM generations.

  3. How does HarnessIR differ from SkCC (Ouyang et al., 2026)? The paper describes SkCC as using a typed intermediate representation for cross-framework skill compilation, which seems closely related to the proposed approach. Could the authors clarify the key methodological differences and consider including SkCC as an additional baseline?

11 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1

We appreciate your recognition of our research contributions and experimental design. Your constructive comments are highly valuable for improving our work. Our point-by-point responses follow.

For W1&Q1: Evaluation design, result reliability, and baseline interpretation.

1. Benchmark Scale and Construction Stability.

Our test set contains 10 distinct source Harnesses, each transferred to 2 target runtimes and tested on 6 scenarios with 3 executions each, yielding 360 target executions per method and backbone configuration. Thus, 10 denotes distinct source Harnesses, not the number of executions. To assess LLM-based construction variability, we added five independent reconstruction rounds for HarnessIR and the strongest baseline, SOT, in the revised Appendix A. Each round repeats the full construction pipeline. HarnessIR achieves a Behavior rate of 49.89% ± 0.32%, compared with 29.72% ± 0.52% for SOT. In the per-source comparison, HarnessIR achieves a higher mean Behavior rate on 8 source Harnesses, ties on 1, and underperforms on 1. These results show that its overall advantage is not attributable to a single construction run.

2. Derivation of Behavioral Obligations.

Behavioral obligations are derived from the source-native execution references of each source Harness under the corresponding test scenarios. Specifically, we combine the source code and execution traces to identify executed actions, their execution contexts, state dependencies, and constraints, and organize the applicable behavioral requirements into five dimensions: State, Control, Handoff, Action Effect, and Termination. All target programs are assessed against the same source-derived obligations and test scenarios.

3. Original Evaluation Protocol and Fairness.

Our evaluation uses source-defined behavioral requirements as a common reference, not method-specific internal representations. The five behavioral dimensions assess Harness behavior preservation from different perspectives. The evaluator receives each method's target programs and execution traces, along with the source-native execution references. The intermediate representations and ontology structures produced during HarnessIR construction are not provided to the evaluator. Thus, the evaluator cannot determine whether target programs use HarnessIR's ontology entities or score them by internal representations. All methods use the same mechanism to record operation requests, arguments, and results, and are assessed against the same source-derived requirements. This ensures a fair and consistent evaluation across all methods. For LLM-based metrics, Task-J uses GLM-5.2 to judge semantic equivalence with identical prompts and configurations across methods. E2E-J jointly requires Task-J, Behavior, and successful execution in the target runtime. We have added this fairness clarification to Section 4 of the revised manuscript.

4. Supplementary Action Consistency Evaluation.

To further validate our findings, we conducted a complementary evaluation directly comparing operations in paired source and target execution records. For all methods, the same collection mechanism records operation requests, arguments, and returned results in a unified format. The paired records are compared using the same GLM-5.2 judge and prompt. The evaluation compares operational semantics, key arguments, operation counts, and execution order, without access to intermediate representations. We use Action Consistency to measure preservation of the source program's operational behavior. Results:

MethodAction Consistency
DCT11.67%
ECT25.00%
SSIR15.28%
SOT42.50%
HarnessIR58.33%

HarnessIR achieves the highest Action Consistency, demonstrating its advantage under this direct comparison of operational behavior. We have added the Action Consistency evaluation to Section 5.5 of the revised manuscript.

5. Explaining the Zero Results for DCT and SSIR.

The zero results of these two methods occur on different metrics. DCT achieves 0% Task success under both backbones: no execution meets all requirements for the correct final response, terminal state, final external state, and action effects. However, its Task-J rates remain non-zero at 8.06% and 10.28%, respectively, indicating that some executions can produce semantically equivalent final responses but fail to complete the full task. SSIR achieves 0% Behavior under both backbones. Its path-oriented design focuses on control-flow extraction and selection without explicitly modeling cross-step state dependencies or action execution constraints. Consequently, it scores 0% on State and Action Effect, yielding 0% overall Behavior despite partial Termination success. Both methods' zero E2E rates follow from the criterion requiring joint success in Task, Behavior, and execution. We have clarified the zero results for DCT and SSIR in Section 5.3 of the revised manuscript.

AuthorsOpenReview ↗

Response to Weakness 2

For W2: E2E performance, IR specification, and compiler analogy.

We further discuss the effectiveness of HarnessIR, clarify its IR definition, and explain its relationship to traditional compiler frameworks.

1. Interpreting E2E Performance.

We acknowledge that the absolute E2E improvements achieved by HarnessIR are modest, and that a substantial proportion of transfers still fail to achieve full end-to-end success. However, E2E is a strict joint metric that requires an execution to simultaneously satisfy Task, Behavior, and successful execution in the target runtime. Therefore, even when a target Harness preserves the execution behavior of the source program, it is still counted as an E2E failure if the final task is not completed. Given that the primary objective of HarnessIR is behavioral preservation across runtimes, we believe its effectiveness should be assessed by considering both Behavior and E2E. In the main experiments, HarnessIR achieves the highest Behavior and E2E rates under both backbones. Specifically, its Behavior rates are 49.44% and 40.56%, compared with 33.33% and 34.72% for the strongest baseline, SOT. Its E2E rates are 11.11% and 13.61%, compared with 10.00% and 11.39% for SOT.

Furthermore, the joint analysis in Appendix F shows that, compared with SOT, HarnessIR increases the proportion of executions that simultaneously satisfy Task and Behavior (12.4% vs. 10.7%) and reduces the proportion of executions that fail on both criteria (48.5% vs. 57.5%). These results indicate that the primary value of HarnessIR lies in improving behavioral preservation across runtimes and thereby increasing the likelihood of achieving complete task success, rather than substantially improving final task success alone. We also acknowledge that there remains considerable room for improving end-to-end performance.

2. IR Definition and Execution Semantics.

HarnessIR is not a sequence of unstructured LLM calls, but a typed intermediate representation built upon a manually designed, predefined Harness ontology. The ontology defines a shared set of entity types, relation types, and attributes. Its core entities, including State, Action, Decision, and Constraint, represent the runtime state, executable operations, runtime decisions, and execution constraints of a Harness, respectively. The complete ontology, including its entity types, relations, and subtypes, is detailed in Appendix B. These predefined structures provide a unified semantic foundation for representing Harnesses across different runtimes.

Specifically, the LLM jointly analyzes the static source code and dynamic execution traces to map and instantiate the execution semantics of the source Harness as concrete entities, relations, and attributes within the predefined ontology. This process produces a structured intermediate representation specific to the source Harness, rather than free-form textual descriptions. The extraction process uses structured JSON outputs constrained by the predefined ontology. We have added the complete output schemas to Appendix G of the revised manuscript to further clarify the structure of the IR.

Furthermore, HarnessIR represents how state and decision conditions determine action activation, as well as the constraints that actions must satisfy, through explicit execution policies. The generated target Harness follows these policies to perform constraint checks and state updates during execution. Thus, HarnessIR provides not only a well-defined structured representation but also execution semantics that guide the behavior of the target Harness.

3. Relationship to Traditional Compiler Frameworks.

Our references to LLVM and MLIR are intended to highlight their shared design principle of using intermediate representations to separate source-program semantics from target-specific implementations, rather than to claim that HarnessIR provides the same capabilities or degree of formalization as mature compiler infrastructures. Our behavioral-preservation evaluation is based on test scenarios; it is not a formal semantic-equivalence proof in the sense of translation validation. HarnessIR is an ontology-guided intermediate representation designed for cross-runtime Agent Harness migration, using structured semantic recovery and execution-policy constraints to support the transfer and preservation of source Harness behavior across heterogeneous runtimes. We have clarified the scope of the compiler analogy in Section 2 of the revised manuscript.

AuthorsOpenReview ↗

Response to Question 2 and Question 3

For Q2: Construction stability and statistical significance.

To assess the variability introduced by LLM-based generation and examine the stability of the construction process, we added five independent reconstruction rounds for HarnessIR and the strongest baseline, SOT, in the revised Appendix A. In each round, HarnessIR is reconstructed from scratch, repeating the entire pipeline, including source semantic recovery and IR construction, target-runtime semantic grounding, and target code generation. SOT is likewise reconstructed from scratch in each round, repeating its complete construction pipeline.

We report the per-round results, means, and sample standard deviations, and compare the two methods using exact two-sided permutation tests. HarnessIR achieves a Behavior rate of 49.89% ± 0.32%, significantly outperforming SOT at 29.72% ± 0.52% (p = 0.0079). HarnessIR also achieves higher E2E and E2E-J rates, at 11.17% ± 0.41% and 22.50% ± 0.90%, respectively, compared with 9.83% ± 0.61% and 19.06% ± 0.32% for SOT (p = 0.0159 and p = 0.0079, respectively).

Furthermore, we report the mean Behavior rate across the five independent reconstruction rounds for each source Harness, together with source-level significance tests (Appendix A, Table 9). HarnessIR achieves a higher mean Behavior rate on 8 source Harnesses, ties on 1, and underperforms on 1. The improvements on Mastra, BeeAI, Strands, and Swarms are statistically significant (p = 0.0079). These results demonstrate that HarnessIR's behavioral preservation advantage remains stable across independent reconstruction rounds, rather than depending on individual LLM generations. The complete per-round results, per-source comparisons, and statistical tests are provided in the revised Appendix A.

For Q3: Differences from SkCC and adapted baseline evaluation.

Although SkCC and HarnessIR both employ intermediate representations (IRs) to support cross-framework transfer, they differ in their research objects and objectives.

1. Research objects and objectives.

A Skill is a reusable package of knowledge and capabilities, typically consisting of natural-language instructions, executable scripts, and related resources. In contrast, an Agent Harness is the runtime infrastructure built around an LLM that manages the model's interactions with the external environment, including state management, tool execution, task orchestration, and control flow. In short, a Skill encapsulates reusable knowledge and capabilities, whereas a Harness manages how these capabilities are invoked and organized during agent execution. SkCC focuses on cross-framework Skill compilation, while HarnessIR addresses the cross-runtime transfer of executable Harnesses, aiming to preserve their runtime state, control logic, and execution behavior. Thus, although both methods use typed IRs, they represent different semantics and pursue different correctness objectives.

2. Applicability of SkCC.

Applying SkCC to Harness migration requires first abstracting the source Harness code and execution traces into a Skill, and then translating the compiled Skill representation back into a target Harness. Neither of these additional stages is within the original scope of SkCC. Moreover, the former may lose information about Harness execution semantics, while the latter may fail to fully recover the target control logic. Consequently, the final migration performance depends not only on SkCC itself but also on these additional adaptation stages, and therefore cannot be regarded as a direct measure of SkCC's native capability.

3. Additional adapted experiment.

To address your suggestion, we conducted a SkCC-adapted experiment. Specifically, we converted the source Harness code and public execution traces into a Skill document, processed it with the original SkCC compiler to construct a typed intermediate representation, and then generated a target-native Harness. The results are as follows:

MethodBehaviorStateTaskTask-JE2EE2E-J
SkCC-adapted26.94%38.89%7.50%27.50%2.50%12.78%

We report this result as a supplementary experiment to illustrate the performance of adapting a Skill compilation method to the Harness migration task.

These are our responses to your comments. Thank you very much for your time and careful consideration. Should any questions remain, we sincerely welcome further feedback and would be happy to provide additional clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.