acceptodds
Under review as a conference paper at ICLR 2027

G-Harness: Architecting and Optimizing Multi-Agent Harnesses via Dynamic Graphs

Abstract

Multi-agent systems (MAS) depend not only on model capabilities, but also on the surrounding harness—the prompts, tools, memory, and control flow that govern agent interaction and execution. In MAS, agent composition, task assignment, tool access, and model binding are tightly coupled and evolve with task progress and runtime conditions, making harness optimization a dynamic system-level problem. Jointly adapting these decisions during execution is challenging because the feasible configuration space changes with the execution state, while the effects of structural changes are often revealed only through delayed task-level feedback. In this paper, we propose G-Harness, a dynamic graph-based framework for architecting and optimizing multi-agent harnesses. G-Harness formulates MAS execution and its surrounding harness as a unified dynamic graph, enabling coupled system organization and execution state to be represented and reasoned about jointly. We develop a task-conditioned graph-edit policy that learns to construct effective task-specific harness graphs and continuously rewrites them during execution, enabling automatic harness optimization and online adaptation based on runtime feedback. We further develop a global–local optimization framework that jointly captures task-level execution utility and the contribution of individual decisions, thereby guiding the policy toward task-relevant structural optimization. Extensive experiments demonstrate three key properties of G-Harness: (1) high-performing, achieving 80.68% Score on Harness-Bench; (2) cost-efficient, achieving a favorable performance-efficiency trade-off compared with existing multi-agent and harness optimization methods; and (3) attack resilient, suffering only a 1.14% performance drop under agent-level prompt attacks.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Harness Engineering, Multi-Agent Systems, Dynamic Graphs, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_xzE3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on dynamically modifying the configuration of multi-agent systems during the task process to utilize runtime conditions for better performance. The paper proposes G-Harness, trying to solve subtasks with different adaptive configurations (e.g., different agent roles and different model binding, etc.).

G-Harness treats a multi-agent system as a graph and dynamically selects active subgraphs to configure agent composition, task assignment, tool access, and model binding across subtask-level state transitions. A Global Harness Graph holds registered agents, tools, memories, candidate edges, and runtime state. An LLM Task Composer turns each task into a static directed acyclic graph. A task-conditioned policy then builds an Active Harness Subgraph and revises it during execution. Edits are drawn from a finite legal set: activate or deactivate components and edges, assign or unassign tasks, bind a model, set a configuration, request an additional registered capability, or commit the graph. The policy is basically a mapping from states to edit actions, with a specially designed encoding for graphs. Specifically, a relation-aware graph Transformer encodes over task, agent, tool, and memory nodes, with a frozen sentence-BERT encoder to produce description embeddings. The decoder is an MLP that maps the state encodings into graph-edit decisions (actions). The policy is trained by cloning trajectories from a teacher LLM, then by trajectory-level Group Relative Policy Optimization, and finally by a local preference stage that compares the committed graph with legal perturbations. After a graph is committed, execution continues until a task-state change or a material change in availability or resources opens another editing round.

The empirical case is made on Harness-Bench, DeepPlanning-Shopping, and ALFWorld. On the authors’ splits, G-Harness reports a Score of 80.68, a Match Score of 87.28, and an average success rate of 89.39 on the three benchmarks, respectively, above the vanilla, multi-agent, and harness baselines they evaluate, at a moderate reported token cost. An agent-level prompt attack on ALFWorld lowers its success rate by 1.14 points, from 89.39 to 88.25. Ablations show drops when the task graph, runtime feedback, online rewriting, GRPO, or local perturbation is removed, and two ALFWorld traces illustrate adding agents after insufficient exploration and rebinding a model after repeated illegal actions.

Critical strengths

The paper is generally well-structured, with a clear motivation and the corresponding solution to address the proposed scenario.

It basically gives insight similar to dynamic algorithm configuration (DAC): during a task process, a dynamic configuration is needed to deal with the different demands of the different stages or subtasks, and the runtime conditions and feedback can be utilized to do so. This insight has relatively extensive application scenarios in practical problems.

The dynamic graph modeling is interesting: selecting an active subgraph as the current configuration to support adaptation to different subtasks.

Experiments on three benchmarks support the excellent performance of the proposed method, as well as extra experiments on cost-performance trade-off and robustness tests.

Critical weaknesses
  1. Strictly speaking, the paper optimizes the structural configuration of the multi-agent system, with the tunable variables being the selection and connection on a pre-registered graph, rather than modifying the source code of the harness itself. Therefore, I think the statement "G-Harness, optimizing multi-agent harnesses" is not that precise. Especially when the policy of the proposed method needs to be retrained if there are new types of graph-edit operations. Moreover, the original Harness-Bench (the main benchmark in the paper's experiments) protocol binds one model and then compares different frozen harnesses, with no re-binding allowed mid-process, to fairly evaluate the pure capabilities of harnesses. However, the protocol of G-Harness is obviously different. It supports switching models midway, making it difficult to purely assess its capabilities as a "harness". This does not mean that the experiments cannot demonstrate the effectiveness of G-Harness. Instead, I think G-Harness is not suitable to be positioned as a method for improving harnesses, but is more suitable to be regarded as an approach for dynamically constructing MAS.
  2. Some formulas in the key formulation of the proposed graph transformer are not clearly described: line 226 Wτ(u)in,cτ(u)inW_{\tau(u)}^{in},c_{\tau(u)}^{in}; line 232 wR(k,p),UR(k,p)w_R^{(k,p)}, U_R^{(k,p)} are not explained.
  3. Many details supporting reproducibility are not specified: the size of the graph transformer, as well as many other hyperparameters in Local Graph Perturbation; the details of the "candidate scorer" (the paper just says it "assigns a scalar score", but what is it? an MLP?). Since the Task Graph is important in the method pipeline, the paper does not provide any examples or details of the Task Graph. In line 1071, the paper claims, "The released trace contains the Task Graph, the pre-edit and post-edit Active Harness snapshots, the accepted graph actions, the environment observations, and the completion evidence", but I cannot find anything about the Task Graph.
  4. The argument for the robustness test is a bit too sketchy. The paper only says "the robustness enabled by task-adaptive harness optimization and local graph perturbation training", but shows no traces that the strategy changed edges and switched models after detecting the lie. Moreover, the paper does not report which agent is attacked (only one of the five agents is attacked), nor whether the malicious output was adopted by other agents. For the single-agent baseline, replacing the system prompt amounts to attacking the sole decision-maker; for G-Harness, it is merely one of up to five roles. These two attack strengths are not equivalent.
  5. Fairness of experimental comparison. For the vanilla model baselines, the paper says, "Reasoning or thinking modes are disabled whenever supported", but does not explain the reason. The original conclusion of DeepPlanning is that the model with maximum reasoning strength enabled is significantly better than models from the same family with reasoning turned off. Moreover, the cost comparison is not quite fair, as only the token cost is reported. However, model sizes vary across different models, and their API prices differ as well. When it comes to costs, small models and large models have substantial gaps in price and latency for the same number of tokens. Some approaches may consume fewer tokens but rely entirely on large, expensive models with massive parameter counts. In that case, I believe the cost of such approaches is also very high.
Questions for authors
  1. Can you explain the reason why reasoning or thinking modes are disabled whenever they are supported for the vanilla model baselines?
  2. Can you provide more details that support reproducibility? For example, the details mentioned in Weaknesses.
  3. Can you provide more analyses of the robustness tests? For example, specifying which agent is attacked (especially the executor).
  4. Can you report cost from a fairer dimension? For example, the api cost?
  5. Can you provide more detailed examples of the specific tasks? For example, the Task Graph.
10 comments
AuthorsOpenReview ↗

Response to Reviewer xzE3 (Part 1 of 5)

Dear Reviewer xzE3,

Thank you for your thoughtful review and constructive feedback. We appreciate your recognition of the motivation, dynamic graph formulation, and empirical effectiveness of G-Harness. We address these concerns and provide additional clarifications and experimental evidence below.

W1: Strictly speaking, the paper optimizes the structural configuration of the multi-agent system…

Response to W1: We thank the reviewer for pointing out this important distinction and for the opportunity to clarify the scope of harness optimization in G-Harness and its relationship to the original Harness-Bench evaluation protocol.

(1) Scope of Harness Optimization. Consistent with existing harness research (e.g., Harness-Bench[1] and HarnessX[2] ), we view a multi-agent harness as the runtime execution and control layer governing the coordination of agents, tools, memory, and model interactions. Accordingly, harness optimization is not limited to source-code modification, but also encompasses the optimization of runtime configurations and control decisions. G-Harness specifically optimizes component activation, interaction topology, task assignments, tool access, and model bindings based on task requirements and execution feedback. Thus, dynamic MAS construction is an integral part of G-Harness's runtime harness optimization, which further extends to continuous structural reconfiguration and learning-based graph-edit policy optimization.

(2) Extensibility of Graph Editing. We acknowledge that introducing fundamentally new graph-edit operation types may require extending and retraining the policy. However, this does not limit G-Harness to predefined harness configurations. While the graph-edit operation types are fixed, the learned policy dynamically determines which components and interactions to modify, when to perform these edits, and how to combine them based on task requirements and runtime feedback. Moreover, existing operations can be applied to newly registered compatible components without changing the action vocabulary.

(3) Benchmark Protocol and Model Rebinding. The original Harness-Bench protocol focuses on evaluating fixed model–harness configurations, while G-Harness investigates runtime-adaptive harness optimization. In our setting, dynamic model rebinding is an integral harness-level control mechanism, allowing the system to adapt when a worker model becomes insufficient for evolving task requirements. To further disentangle the benefits of structural adaptation from heterogeneous model capabilities, we additionally evaluate G-Harness using GLM-5.3-Flash exclusively, without cross-model switching (Table 1). Under this setting, G-Harness achieves 77.84% on Harness-Bench and 85.08% on ALFWorld, outperforming the strongest fixed-worker baselines by 3.36 and 1.86 percentage points, respectively. These results demonstrate that G-Harness's performance advantages are not solely attributable to model switching, but also arise from adaptive harness configuration and structural optimization.

Table 1. Local Optimization Under a Shared Worker Model

MethodWorkerModelHarness-BenchScore(%)↑ALFWorldAvg.SR(%)↑
Harness-R1GLM-5.3-Flash64.7283.22
HarnessXGLM-5.3-Flash74.4879.07
Meta-HarnessGLM-5.3-Flash72.4782.02
G-Harness(GRPO-only,Budget-matched)GLM-5.3-Flash69.8579.64
G-Harness(Global–Local)GLM-5.3-Flash77.8485.08

Weakness 2. Some formulas in the key formulation of the proposed graph transformer are not clearly described: line 226, Wτ(u)inW_{\tau(u)}^{in} and cτ(u)inc_{\tau(u)}^{in}; line 232, wR(k,p)w_R^{(k,p)} and UR(k,p)U_R^{(k,p)} are not explained.

Response to W2: We thank the reviewer for identifying these missing definitions. We clarify the four parameters as follows:

  • Wτ(u)inW_{\tau(u)}^{\mathrm{in}} and cτ(u)inc_{\tau(u)}^{\mathrm{in}} (Line 226) are the learnable type-specific input projection matrix and bias vector, respectively, which map the concatenated description embedding and node features into the shared hidden representation space. Here, τ(u)\tau(u) denotes the node type (task, agent, tool, or memory).
  • wR(k,p)w_R^{(k,p)} (Line 232) is a learnable relation-scoring vector for attention head pp in layer kk, projecting the relation features ρuv\rho_{uv} into a scalar attention bias.
  • UR(k,p)U_R^{(k,p)} (Line 234) is a learnable relation-projection matrix that maps ρuv\rho_{uv} into the attention value space, incorporating structural and relational information into message aggregation.

These parameters enable type-aware node encoding and relation-aware attention, allowing the Graph Transformer to distinguish heterogeneous components and their interactions. We will explicitly define these parameters in the camera-ready version.

AuthorsOpenReview ↗

Response to Reviewer xzE3 (Part 2 of 5)

W3: Many details supporting reproducibility are not specified: the size of the graph transformer, as well as many other hyperparameters in Local Graph Perturbation; the details of the "candidate scorer" (the paper just says it "assigns a scalar score", but what is it? an MLP?)… but I cannot find anything about the Task Graph. Q2: Can you provide more details that support reproducibility? For example, the details mentioned in Weaknesses. Q2: Can you provide more details that support reproducibility? For example, the details mentioned in Weaknesses. Q5: Can you provide more detailed examples of the specific tasks? For example, the Task Graph.

Response to W3, Q2 & Q5: We thank the reviewer for highlighting the need for more detailed implementation specifications and concrete Task Graph examples.

(1) Architecture and Training Details. Our original submission provides the Task Composer and training configurations in Appendix A.3 (Reproducibility Details), together with the graph-edit space and local perturbation procedure in Appendix B.1–B.2. We further provide the missing implementation settings, particularly the hyperparameters of Local Graph Perturbation Optimization, in Table 2. For further implementation details and experimental configurations, please refer to our anonymized code repository: https://anonymous.4open.science/r/ICLR-G-Harness-9CFF/. We will incorporate these additional settings into the camera-ready appendix.

(2) Candidate Scorer Implementation. The candidate scorer is a two-layer MLP (640 → 128 → 1) with GELU activation. It concatenates the 128-dimensional decoder state, the selected operation embedding, and up to three role-specific argument representations. Node arguments use Graph Transformer embeddings, while non-node arguments use projected text embeddings. The MLP assigns a scalar score to each Validator-approved argument tuple, and a softmax over legal candidates produces the conditional argument distribution.

Table 2. Additional Implementation and Training Configurations

CategoryHyperparameterValue
Graph PolicyHidden Dimension128
Graph Transformer Layers2
Attention Heads4
Dropout0.0
Reward NormalizationExecution-cost Component Weights(1.0, 1.0, 1.0, 1.0, 1.0)
Execution-cost Normalization Scales(96K tokens, 50 calls, 600s, 1 CNY, 4 expansions)
Graph-cost Component Weights(1.0, 0.2, 0.5)
Graph-cost Normalization Scales(16 nodes, 32 edges, 32 edits)
Graph EditingMaximum Graph-edit Budget32
Local Graph PerturbationDPO Scale (κ)(\kappa)0.1
Local-loss Weight (λlocal)(\lambda_{\mathrm{local}})0.1
Maximum Perturbed Alternatives per Branch Point2
Maximum Edits per Perturbation3
Perturbation Search Frontier Limit64
Maximum Refinement Updates (Harness-Bench / ALFWorld)32 / 125

(3) Task Graph Construction and Examples. We thank the reviewer for requesting concrete Task Graph examples. The Task Graph is formally defined in Section 3 (Task Graph), with its construction and validation procedures detailed in Appendix A.3 (Reproducibility Details). To further clarify its construction, we provide representative examples from both ALFWorld and Harness-Bench in the examples/ directory of our anonymized code repository (https://anonymous.4open.science/r/ICLR-G-Harness-9CFF/). Specifically, each example includes a machine-readable Task Graph (task_graph.json) and its visualization (task_graph.svg), together with Active Harness snapshots, graph-edit decisions, and execution traces. These examples illustrate how task-specific dependencies are represented and how the harness adapts during execution.

AuthorsOpenReview ↗

Response to Reviewer xzE3 (Part 3 of 5)

W4: The argument for the robustness test is a bit too sketchy. The paper only says "the robustness enabled by task-adaptive harness optimization and local graph perturbation training", but shows no… Q3: Can you provide more analyses of the robustness tests? For example, specifying which agent is attacked (especially the executor).

Response to W4 & Q3: We thank the reviewer for highlighting the need for a more detailed robustness analysis, particularly regarding attack-target specification, evaluation fairness, and concrete evidence of runtime graph adaptation. We address these concerns through additional controlled comparisons, robustness ablations, and an Executor-targeted execution trace.

(1) Attack Protocol and Executor-Targeted Evaluation. Following G-Designer[1], our original robustness evaluation applies a system-prompt attack to one eligible task-performing agent, selected deterministically using the task identifier and random seed 42. To address the concern that compromising the sole decision-maker in a single-agent system differs from attacking an arbitrary role in a multi-agent system, we additionally conduct Executor-targeted attacks under a shared GLM-5.3-Flash worker configuration (Table 3). Specifically, we target the sole Executor in Vanilla, the task-executing agent in LEMON, and the Executor in G-Harness. Under this setting, Vanilla and LEMON suffer success-rate drops of 13.63 and 10.60 percentage points, respectively, whereas G-Harness exhibits a smaller drop of 6.01 points, retaining 79.27% attacked SR. These results provide a more direct assessment of robustness when the critical execution role is compromised, rather than relying solely on random-role attacks.

(2) Robustness Mechanism Ablation. To investigate whether robustness arises from the proposed mechanisms rather than multi-agent redundancy alone, we further evaluate two ablations under the original random-role attack protocol (Table 4). Removing Online Rewriting reduces attacked SR from 88.25% to 68.83%, with the SR drop increasing from 1.14 to 10.66 points. Removing Local Graph Perturbation Optimization yields 70.68% attacked SR and a 6.94-point degradation. These results support the contributions of both task-adaptive runtime graph rewriting and local graph perturbation optimization to adversarial robustness.

(3) Attack Propagation and Runtime Graph Reconfiguration. To provide concrete evidence of runtime adaptation under adversarial attacks, we include an Executor-targeted ALFWorld case study in examples/Example_executor_attack.json of our anonymized code repository (https://anonymous.4open.science/r/ICLR-G-Harness-9CFF/). In this example, the attacked Executor produces misleading information that is received but not adopted by the Reasoner. Based on contradictory environment feedback, G-Harness deactivates the affected communication edge, reassigns the localization task, and switches the Reasoner's model from GLM-5.3-Flash to Qwen3.7-Plus, ultimately completing the task successfully. The trace documents the attack propagation and corresponding graph-edit actions.

Table 3. Robustness Under Random and Executor-Targeted Agent Attacks on ALFWorld

Attack SettingMethodWorker ModelAttacked RoleClean SR (%) ↑Attacked SR (%) ↑SR Drop (pp) ↓
Original Random-role AttackG-HarnessOriginal Model PoolRandom Eligible Agent89.3988.251.14
Shared-worker Executor AttackVanilla AgentGLM-5.3-FlashSole Executor58.9645.3313.63
LEMONGLM-5.3-FlashTask-executing Agent61.8451.2410.60
G-HarnessGLM-5.3-FlashExecutor85.2879.276.01

Table 4. Robustness Ablation Under Random-role Agent Attacks on ALFWorld.

MethodClean SR (%) ↑Attacked SR (%) ↑SR Drop (pp) ↓
w/o Online Rewriting79.4968.8310.66
w/o Local Graph Perturbation Optimization77.6270.686.94
G-Harness89.3988.251.14
Reviewer_AtDROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes G-Harness, a framework that represents agents, tools, memory, and execution state as dynamic graphs. A learned graph-edit policy constructs and rewrites task-specific harnesses, using supervised initialization, GRPO, and local graph-perturbation preferences. Experiments report improved performance on Harness-Bench, DeepPlanning-Shopping, and ALFWorld, with ablations supporting the benefits of online adaptation and policy optimization.

Critical strengths

Unified Runtime Modeling: The framework jointly optimizes agent assignment, model binding, and capability access through constrained graph edits. This provides a concrete mechanism for adapting the harness during execution, rather than only choosing its initial topology.

Critical weaknesses

Limited Reliability and Attribution: Harness-Bench and DeepPlanning-Shopping use only 21 and 24 test tasks, without repeated-run uncertainty estimates. Moreover, harness baselines use GLM-5.3-Flash while G-Harness can route across five worker models. The local-optimization ablation also lacks a matched-training-budget GRPO-only control. These gaps limit confidence in the rankings and in attributing improvements to the proposed harness and local supervision. Incomplete Reproducibility and Cost Analysis: Important settings are unspecified, including graph-policy architecture dimensions, execution-cost weights and normalization scales, graph-edit budget, DPO scale, local-loss weight, and perturbation/refinement budgets. Efficiency comparisons mainly report inference tokens, leaving the training cost of branch evaluations and deployment latency unclear. These omissions hinder replication and assessment of the performance–cost trade-off.

Questions for authors
  1. Can you report uncertainty across independent runs and paired per-task results? Do the Harness-Bench rankings persist across the available group-aware folds?
  2. Does the advantage persist with a shared GLM-5.3-Flash worker model and with local refinement compared against GRPO-only refinement under matched training and branch-evaluation budgets?
  3. Could you provide the missing architecture, reward-normalization, graph-edit, and local-training settings, together with separate training-cost and deployment-latency measurements?
6 comments
AuthorsOpenReview ↗

Response to Reviewer AtDR (Part 1 of 3)

Dear Reviewer AtDR,

Thank you for your careful review and constructive feedback. We appreciate your recognition of G-Harness's unified runtime modeling and adaptive graph-editing framework. We address your concerns regarding experimental reliability, fairness of comparisons, reproducibility, and computational costs below.

Q1: Can you report uncertainty across independent runs and paired per-task results? Do the Harness-Bench rankings persist across the available group-aware folds?

Response to Q1: Thank you for raising this important concern. We conducted additional evaluations to assess performance uncertainty across independent runs, consistency across individual tasks, and ranking stability across group-aware folds.

(1) Reliability Across Independent Runs. We conducted three additional independent runs for G-Harness and six representative baselines on Harness-Bench and ALFWorld. Methods requiring training were independently retrained with different random seeds, while training-free methods were independently evaluated. The original single-run results were excluded from these statistics. As shown in Table 1, G-Harness achieves the highest mean performance on both benchmarks, obtaining 80.34 ± 0.87% on Harness-Bench and 88.10 ± 0.92% on ALFWorld (mean ± standard deviation). These results support the stability of G-Harness's performance advantage across independent runs.

Table 1. Performance Comparison and Stability Across Independent Runs

MethodHarness-Bench Score (%) ↑ALFWorld Avg. SR (%) ↑
Multi-Agent Systems
LEMON54.32± 1.1462.25± 1.21
AFlow71.23 ± 0.8965.53± 0.87
MAS-Orchestra62.07± 1.2348.27± 0.55
Harness-based Methods
Harness-R165.62 ± 0.9283.62± 0.90
HarnessX75.38± 1.3479.87± 1.50
Meta-Harness71.97± 0.7982.19± 0.54
G-Harness80.34 ± 0.8788.10 ± 0.92

(2) Paired Per-Task Evaluation. To investigate whether the improvements are driven by a small number of favorable tasks, we conducted a paired comparison between G-Harness and Harness-R1 on the same 21 Harness-Bench test tasks. As shown in Table 2, G-Harness outperforms Harness-R1 on 19 of 21 tasks (90.5%), achieving an average improvement of 14.72 percentage points (80.34% vs. 65.62%), with a 95% paired bootstrap confidence interval of [11.13, 18.08] percentage points. These results indicate that the improvement is broadly distributed across tasks rather than being driven solely by a few favorable instances.

Table 2. Paired Per-Task Comparison on Harness-Bench

TaskHarness-R1Score(%)G-HarnessScore(%)Difference(pp)
Task0144.6361.34+16.71
Task0257.4078.64+21.24
Task0371.6881.92+10.24
Task0482.1690.39+8.23
Task0564.0083.77+19.77
Task0670.5689.81+19.25
Task0750.8173.64+22.83
Task0884.9479.33−5.61
Task0977.8990.71+12.82
Task1056.5170.01+13.50
Task1162.7580.69+17.94
Task1236.9066.34+29.44
Task1378.9984.18+5.19
Task1467.2683.10+15.84
Task1572.5985.27+12.68
Task1653.8776.49+22.62
Task1792.8992.84−0.05
Task1862.8382.85+20.02
Task1952.5075.54+23.04
Task2076.5084.56+8.06
Task2160.3675.72+15.36
Mean65.6280.34+14.72

(3) Stability Across Group-Aware Folds. As described in Appendix A.1 (Datasets and Evaluation Protocol), our original experimental protocol constructs a group-aware five-fold split of the 106 Harness-Bench tasks using seed 42 and a prompt-similarity threshold of 0.82 to group near-duplicate tasks. While the main experiments use Fold 2 as the fixed evaluation split, we further evaluated G-Harness and Harness-R1 across all five folds, maintaining three folds for training, one for validation, and one for testing. As shown in Table 3, G-Harness consistently outperforms Harness-R1 across all five folds, with improvements ranging from 11.63 to 20.93 percentage points and an average improvement of 15.67 percentage points. The relative ranking between the two methods remains unchanged across all folds.

Taken together, these complementary evaluations demonstrate that G-Harness's performance advantage remains consistent across independent random seeds, individual test tasks, and group-aware evaluation splits, strengthening the reliability of our reported findings. We will include these additional experimental results and analyses in the camera-ready version.

Table 3. Group-Aware Five-Fold Evaluation on Harness-Bench

FoldTestTasksHarness-R1Score(%)↑G-HarnessScore(%)↑Improvement(pp)
Fold12265.1276.84+11.72
Fold2(Original)2164.7280.68+15.96
Fold32162.4383.36+20.93
Fold42166.5278.15+11.63
Fold52163.8781.97+18.10
Mean—64.5380.20+15.67
AuthorsOpenReview ↗

Response to Reviewer AtDR (Part 2 of 3)

Q2: Does the advantage persist with a shared GLM-5.3-Flash worker model and with local refinement compared against GRPO-only refinement under matched training and branch-evaluation budgets?

Response to Q2: Thank you for this important question. We address both concerns through controlled comparisons that examine the effectiveness of G-Harness under a shared worker model and matched training-interaction budgets.

(1) Performance Under a Shared Worker Model. To eliminate the potential advantage of heterogeneous model routing, we restrict all G-Harness worker agents to GLM-5.3-Flash, matching the worker model used by the harness-based baselines. As shown in Table 4, G-Harness (Global–Local) achieves 77.84% on Harness-Bench and 85.08% on ALFWorld, outperforming the strongest baselines by 3.36 and 1.86 percentage points, respectively. Furthermore, under the same worker model, Global–Local optimization outperforms budget-matched GRPO-only refinement by 7.99 and 5.44 percentage points. These results support the effectiveness of G-Harness beyond heterogeneous worker-model capabilities and additional GRPO refinement.

(2) Effectiveness Beyond Additional Training. To isolate the benefits of local refinement from additional sampling, we compare G-Harness against GRPO-only under matched training-interaction budgets and optimizer updates. As shown in Tables 5 and 6, G-Harness achieves improvements of 7.98 percentage points on Harness-Bench and 6.29 percentage points on ALFWorld, demonstrating that local optimization provides benefits beyond additional GRPO sampling.

These comparisons support the effectiveness of G-Harness under a shared worker backbone and demonstrate that Local Graph Perturbation Optimization provides benefits beyond additional GRPO sampling. We will incorporate these experimental results and implementation details into the camera-ready version.

Table 4. Local Optimization Under a Shared Worker Model

MethodWorker ModelHarness-Bench Score (%) ↑ALFWorld Avg. SR (%) ↑
Harness-R1GLM-5.3-Flash64.7283.22
HarnessXGLM-5.3-Flash74.4879.07
Meta-HarnessGLM-5.3-Flash72.4782.02
G-Harness (GRPO-only, Budget-matched)GLM-5.3-Flash69.8579.64
G-Harness (Global–Local)GLM-5.3-Flash77.8485.08

Table 5. Training-Budget-Matched Comparison on Harness-Bench

MethodGRPO RolloutsBranch Cont.Env. StepsTrain Tokens (M)GRPO UpdatesDPO UpdatesTotal UpdatesScore (%) ↑
GRPO-only (Original)768030,720103.834804863.43
GRPO-only (Budget-matched)1,024040,420119.106406472.70
G-Harness (Full)89625640,960120.405686480.68

Table 6. Training-Budget-Matched Comparison on ALFWorld

MethodGRPO RolloutsBranch Cont.Env. StepsTrain Tokens (M)GRPO UpdatesDPO UpdatesTotal UpdatesAvg. SR (%) ↑
GRPO-only (Original)3,200076,800166.40200020077.62
GRPO-only (Budget-matched)3,712088,240191.65232023283.10
G-Harness (Full)3,45651289,088193.022161623289.39
AuthorsOpenReview ↗

Response to Reviewer AtDR (Part 3 of 3)

Q3: Could you provide the missing architecture, reward-normalization, graph-edit, and local-training settings, together with separate training-cost and deployment-latency measurements?

Response to Q3: Thank you for highlighting the need for more detailed implementation settings and cost analysis. We clarify the existing configurations and provide additional information to improve reproducibility and transparency.

(1) Implementation and Training Settings. Our original submission already provides key implementation details in Appendix A (Experimental Details), including the reward formulation, cost coefficients, SFT initialization, and GRPO training settings. Appendix B.1 (Graph Edit Space) and Appendix B.2 (Local Perturbation Proposal) further specify the graph-edit operations and local perturbation procedure. To address the remaining omissions, we provide additional hyperparameter configurations in Table 7.

Table 7. Additional Implementation and Training Configurations

CategoryHyperparameterValue
Graph PolicyHidden Dimension128
Graph Transformer Layers2
Attention Heads4
Dropout0.0
Reward NormalizationExecution-cost Component Weights(1.0, 1.0, 1.0, 1.0, 1.0)
Execution-cost Normalization Scales(96K tokens, 50 calls, 600s, 1 CNY, 4 expansions)
Graph-cost Component Weights(1.0, 0.2, 0.5)
Graph-cost Normalization Scales(16 nodes, 32 edges, 32 edits)
Graph EditingMaximum Graph-edit Budget32
Local RefinementDPO Scale (κ)(\kappa)0.1
Local-loss Weight (λlocal)(\lambda_{\mathrm{local}})0.1
Maximum Perturbed Alternatives per Branch Point2
Maximum Edits per Perturbation3
Perturbation Search Frontier Limit64
Maximum Refinement Updates (Harness-Bench / ALFWorld)32 / 125

For further implementation details, including the graph-policy architecture, training procedures, and experimental configurations, please refer to our anonymized code repository: https://anonymous.4open.science/r/ICLR-G-Harness-9CFF/. We will incorporate these additional settings into the revised appendix.

(2) Training Cost and Deployment Latency. To clarify the computational overhead, we further analyze the training and execution logs retained from our original experiments, with the results summarized in Table 8. As shown in Table 8(a), G-Harness performs 294 and 594 branch evaluations on Harness-Bench and ALFWorld, consuming 75.10M and 33.16M training tokens, respectively, including 39.53M and 17.68M tokens associated with local refinement. For deployment, Table 8(b) reports average policy-decision latencies of 227.4 ms and 193.1 ms, with cumulative policy overheads of 6.00 s and 2.75 s per task, accounting for approximately 7.02% and 8.29% of end-to-end execution time, respectively. These retrospectively aggregated statistics provide a transparent and traceable account of G-Harness's training costs and deployment overhead.

Table 8. Training Cost and Deployment Latency of G-Harness (Preliminary)

(a) Training Cost

BenchmarkGRPO RolloutsBranch EvaluationsTraining Tokens (M)Branch-related Tokens (M)Cumulative Branch Execution Time (h)
Harness-Bench51229475.1039.539.71
ALFWorld25659433.1617.682.86

(b) Deployment Latency

BenchmarkPolicy Latency / Decision (ms) ↓Policy Overhead / Task (s) ↓End-to-End Latency / Task (s) ↓
Harness-Bench227.46.0085.50
ALFWorld193.12.7533.17
Reviewer_9qd9OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This is a relatively complete Multi Agent Harness Optimization work. The main innovation of the paper lies in unifying the organizational structure, task allocation, model binding, tool access, and runtime state of multi-agent systems into a dynamic computing graph, and implementing runtime adjustments through the learned Graph edit Policy.

Critical strengths

Unified graph-edit policy for runtime adaptation. A key strength of G-Harness is its unified policy for both initial harness construction and runtime reconfiguration. Rather than committing to a fixed multi-agent architecture before execution, the policy can activate additional agents, rebind models, and reassign tasks in response to evolving task states and runtime feedback. This moves beyond static architecture selection by treating system organization as a sequence of state-conditioned structural decisions, providing a flexible mechanism for adapting multi-agent coordination throughout execution.

Critical weaknesses

None

Questions for authors
  1. Novelty beyond existing graph-based multi-agent design.How does G-Harness differ conceptually and algorithmically from extending G-Designer to incorporate task assignment, model binding, and tool access, together with runtime feedback? Which components constitute substantive methodological advances beyond expanding the graph representation and action space?
  2. Distinctiveness of the optimization framework and its credit-assignment mechanism. Given prior work such as Harness-R1 on learning harness-editing policies through reinforcement learning and execution feedback, what is the principal methodological contribution of G-Harness’s global–local optimization? In particular, does Local Graph Perturbation Optimization provide a finer-grained credit-assignment mechanism, or primarily generate additional preference supervision through extra environment interactions? Since its preferences compare potentially multi-edit sequences based on downstream execution utility, how precisely can it attribute improvements to individual structural decisions? A comparison under matched training-interaction budgets would help distinguish the proposed mechanism’s benefits from those of additional sampling.
  3. Specific benefits of the dynamic-graph formulation. Beyond providing a unified representation of system components and execution states, what specific capabilities or inductive biases does the dynamic-graph formulation introduce relative to a conventional state-conditioned sequential decision policy or architecture-search approach with runtime feedback? Could the authors provide evidence that the improvements arise specifically from graph-structured modeling, rather than from access to richer state information, a broader action space, or additional policy optimization?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 9qd9 (Part 1 of 3)

Dear Reviewer 9qd9,

We sincerely thank you for your thoughtful review and constructive feedback. We greatly appreciate your recognition of our unified graph-edit policy and its ability to support dynamic runtime adaptation. We address your questions and clarify our contributions below.

Q1: Novelty beyond existing graph-based multi-agent design.How does G-Harness differ conceptually and algorithmically from extending G-Designer to incorporate task assignment, model binding, and tool access, together with runtime feedback? Which components constitute substantive methodological advances beyond expanding the graph representation and action space?

Response to Q1: Thank you for this insightful question. While both G-Designer and G-Harness employ graph-based modeling, their optimization objectives and algorithmic mechanisms differ fundamentally.

(1) From topology generation to closed-loop harness control. G-Designer learns a task-conditioned communication topology to guide multi-agent collaboration. In contrast, G-Harness formulates harness optimization as a runtime-conditioned sequential decision process, where a unified policy continuously constructs and rewrites the execution graph based on evolving task states and environmental feedback.

(2) From communication structure to executable configuration. While G-Designer incorporates model and tool information as agent attributes, its learned decisions primarily concern communication connectivity. G-Harness instead jointly optimizes task assignments, model bindings, capability access, and interaction channels through a hierarchical operation–argument decoder, with state-dependent legality constraints ensuring executable and resource-feasible configurations.

(3) From topology-level optimization to global–local harness optimization. G-Designer optimizes communication topologies through policy gradients with anchor and sparsity regularization. In contrast, G-Harness learns a sequential graph-edit policy that jointly considers task performance, execution costs, and structural complexity. Global GRPO optimizes complete graph-edit trajectories, while local graph perturbation compares alternative configurations from shared runtime states to provide targeted DPO preference supervision.

Thus, G-Harness advances beyond communication topology design toward a unified framework for learning how multi-agent harnesses should be constructed, executed, and reconfigured throughout task execution.

AuthorsOpenReview ↗

Response to Reviewer 9qd9 (Part 2 of 3)

Q2: Distinctiveness of the optimization framework and its credit-assignment mechanism. Given prior work such as Harness-R1 on learning harness-editing policies through reinforcement learning and execution feedback, what is the principal methodological contribution of G-Harness’s global–local optimization? …

Response to Q2: Thank you for this insightful question. The principal contribution of our global–local optimization is to provide finer-grained credit assignment for structural decisions through runtime-conditioned counterfactual comparisons.

(1) Methodological distinction. Harness-R1 learns executable harness patches primarily through trajectory-level reinforcement learning and execution feedback. In contrast, G-Harness combines global GRPO with Local Graph Perturbation Optimization. While global GRPO optimizes complete execution trajectories, local optimization explicitly evaluates which structural changes, including agent components, interaction edges, task assignments, and model bindings, contribute to improved execution utility.

(2) Role of Local Graph Perturbation Optimization. Our method provides more localized, structure-sensitive preference supervision than trajectory-level GRPO, rather than explicitly estimating the independent contribution of each atomic graph edit. By introducing targeted graph perturbations and comparing their downstream execution utilities from the same restored runtime state, it provides direct comparative feedback on alternative structural configurations. For single-edit perturbations, the comparison reflects the conditional effect of the corresponding structural change; for multi-edit perturbations, it captures the joint effect of interacting decisions. These comparisons are converted into DPO preferences, allowing the policy to learn from targeted structural feedback rather than relying solely on trajectory-level rewards.

(3) Effectiveness Beyond Additional Training. To distinguish the benefits of local optimization from additional sampling, we conduct training-budget-matched comparisons with identical optimizer updates and comparable environment interactions and training tokens. As shown in Tables 1 and 2 below, G-Harness outperforms budget-matched GRPO-only by 7.98 percentage points on Harness-Bench (80.68 vs. 72.70) and 6.29 percentage points on ALFWorld (89.39 vs. 83.10). These results demonstrate that the improvements cannot be explained solely by additional training interactions, supporting the effectiveness of our finer-grained structural supervision.

Table 1. Training-Budget-Matched Comparison on Harness-Bench

MethodGRPO RolloutsBranch Cont.Env. StepsTrain Tokens (M)GRPO UpdatesDPO UpdatesTotal UpdatesScore (%) ↑
GRPO-only (Original)768030,720103.834804863.43
GRPO-only (Budget-matched)1,024040,420119.106406472.70
G-Harness (Full)89625640,960120.405686480.68

Table 2. Training-Budget-Matched Comparison on ALFWorld

MethodGRPO RolloutsBranch Cont.Env. StepsTrain Tokens (M)GRPO UpdatesDPO UpdatesTotal UpdatesAvg. SR (%) ↑
GRPO-only (Original)3,200076,800166.40200020077.62
GRPO-only (Budget-matched)3,712088,240191.65232023283.10
G-Harness (Full)3,45651289,088193.022161623289.39
AuthorsOpenReview ↗

Response to Reviewer 9qd9 (Part 3 of 3)

Q3: Specific benefits of the dynamic-graph formulation. Beyond providing a unified representation of system components and execution states, what specific capabilities or inductive biases does the dynamic-graph formulation introduce relative to a conventional state-conditioned sequential decision policy or architecture-search approach with runtime feedback? Could the authors provide evidence that the improvements arise specifically from graph-structured modeling, rather than from access to richer state information, a broader action space, or additional policy optimization?

Response to Q3: Thank you for this important question. The key benefit of our dynamic-graph formulation is that it introduces explicit relational inductive biases for modeling structural dependencies and guiding runtime harness reconfiguration, rather than merely expanding the state information or action space.

(1) Graph-Structured Inductive Bias. Unlike conventional state-conditioned sequential policies, G-Harness explicitly represents task dependencies, component interactions, and active harness configurations through typed, directed relations. Its relation-aware Graph Transformer incorporates these relations into attention and message passing, enabling the policy to capture dependencies among agents, tools, memories, and tasks. Furthermore, the graph-edit policy supports coordinated structural decisions under explicit validity constraints, allowing the harness to adapt to evolving execution states rather than relying on static architecture selection.

(2) Effectiveness of Graph-Structured Modeling. To isolate the contribution of graph-structured modeling, we conduct an additional comparison against a flat-state policy that replaces the relation-aware Graph Transformer with a standard Transformer while preserving identical state information, action spaces, optimization objectives, and training budgets. As shown in Table 3 below, removing graph-structured modeling reduces the Harness-Bench Score from 80.34% to 68.42% and ALFWorld Avg. SR from 88.10% to 74.16%. These results demonstrate that the improvements cannot be attributed solely to richer state information, broader action spaces, or additional policy optimization, supporting the effectiveness of explicit graph-structured modeling for dynamic harness optimization.

Table 3. Effect of Graph-Structured Modeling

MethodHarness-Bench Score (%) ↑ALFWorld Avg. SR (%) ↑
w/o Graph-structured Modeling68.42± 1.4674.16± 1.32
G-Harness (Full)80.34± 0.8788.10± 0.92
AuthorsOpenReview ↗

Looking forward to your reply!

Dear Reviewer 9qd9,

We sincerely thank you again for your positive evaluation and constructive feedback. We greatly appreciate your recognition of our work and the time you have devoted to reviewing our submission.

As the review discussion period is approaching its end, we would like to kindly follow up on our previous responses. We have provided detailed clarifications and additional experimental evidence, including training-budget-matched comparisons and further evaluations demonstrating the benefits of graph-structured modeling.

We hope these clarifications and additional results have helped address your questions and strengthen your confidence in our contributions and experimental findings. We would be very grateful if you could review our responses and let us know whether any concerns remain before the discussion period concludes.

Thank you once again for your valuable time, consideration, and support.

Warmest regards,

The Authors

Reviewer_Nyo1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes G-Harness, a framework for dynamically constructing and optimizing multi-agent harnesses through graph-based representations and learned runtime orchestration. It represents available resources, task dependencies, and active agent configurations using interconnected graphs, and employs a graph Transformer policy to perform constrained edits involving agent selection, task assignment, communication, and model routing.

Critical strengths
  • The framework connects task assignment, model selection, capability access, and communication with runtime observations.
  • The graph encoder, constrained decoder, supervised initialization, and policy-optimization stages are specified.
  • Useful controlled analyses.
Critical weaknesses
  • Small evaluation partitions and a budget inconsistency limit.
  • Several comparisons conflate representation, information availability, model access, and optimization budget.
Questions for authors
  1. Does the no-local-optimization variant receive the same number of training interactions and optimizer updates as the full method? Please report the actual rollout, continuation, and update counts for both configurations.
  2. Are the structural and feedback ablations retrained under their modified configurations? Please specify which task information, observations, and completion signals remain available to the policy, workers, and runtime.
  3. Is the reported success rate averaged over individual tasks or equally across task categories? Please provide per-category success counts and denominators to clarify the aggregation.
  4. How can Table 5’s average consumption of 329.2K tokens be reconciled with the stated 256K per-task budget? Please clarify whether the execution budget, token-accounting convention, or maximum number of active agents changes in this experiment.
  5. What evidence attributes the observed attack resilience specifically to local perturbation training or online rewriting, rather than model choice or deterministic validation? Evaluating the corresponding ablations under identical attack conditions would help isolate these contributions.
  6. How does G-Harness compare with strong single-agent systems equipped with comparable tools, memory, and reasoning budgets? Where applicable, comparisons with established agent harnesses such as Claude Code or Codex would help establish whether multi-agent orchestration provides benefits beyond stronger single-agent execution.
  7. Since reasoning modes are disabled or restricted for vanilla models, it remains unclear whether the reported improvements persist against reasoning-enabled baselines under comparable inference budgets. This also raises questions about how G-Harness scales as underlying models become stronger and require less explicit orchestration.
  8. G-Harness uses a high-reasoning-effort DeepSeek-V4-Pro Task Composer and supports heterogeneous worker routing, whereas the harness-based baselines rely on GLM-5.3-Flash as their task-facing model. These differences confound the attribution of performance gains to harness optimization; comparisons with matched worker backbones, task decomposition, and reasoning budgets would help isolate the proposed method's contribution.
  9. The proposed global–local optimization appears to combine trajectory-level GRPO with configuration-level DPO, rather than explicitly estimating the contribution of individual graph edits. How does the method distinguish fine-grained credit assignment from preference learning over complete edit sequences, and can the authors disentangle the benefits of local preference supervision from additional counterfactual rollouts and continued GRPO training?
9 comments
AuthorsOpenReview ↗

Response to Reviewer Nyo1 (Part 1 of 5)

Dear Reviewer Nyo1,

Thank you for revisiting our submission and for your detailed and constructive feedback. We appreciate your careful consideration of our work and the opportunity to clarify the concerns regarding experimental fairness, optimization procedures, and evaluation protocols. We address your questions point by point below.

Q1: Does the no-local-optimization variant receive the same number of training interactions and optimizer updates as the full method? Please report the actual rollout, continuation, and update counts for both configurations.

Response to Q1: We thank the reviewer for raising this important concern. To isolate the contribution of Local Graph Perturbation Optimization, we conduct additional controlled experiments under matched training budgets and worker-model configurations.

Tables 1 and 2 report the actual GRPO rollouts, branch continuations, environment steps, training tokens, and optimizer updates. Under closely matched training budgets and identical optimizer update counts, G-Harness outperforms the budget-matched GRPO-only baseline by 7.98 percentage points on Harness-Bench (80.68% vs. 72.70%) and 6.29 percentage points on ALFWorld (89.39% vs. 83.10%). Furthermore, Table 3 fixes the worker model to GLM-5.3-Flash across all compared methods. Under this controlled setting, Global–Local Optimization consistently outperforms budget-matched GRPO-only by 7.99 and 5.44 percentage points on Harness-Bench and ALFWorld, respectively, while also outperforming Harness-R1, HarnessX, and Meta-Harness.

Together, these results provide evidence that the improvements stem from Local Graph Perturbation Optimization, rather than merely additional training expenditure or stronger worker models.

Table 1. Training-Budget-Matched Comparison on Harness-Bench

MethodGRPO RolloutsBranch Cont.Env. StepsTrain Tokens (M)GRPO UpdatesDPO UpdatesTotal UpdatesScore (%) ↑
GRPO-only (Original)768030,720103.834804863.43
GRPO-only (Budget-matched)1,024040,420119.106406472.70
G-Harness (Full)89625640,960120.405686480.68

Table 2. Training-Budget-Matched Comparison on ALFWorld

MethodGRPO RolloutsBranch Cont.Env. StepsTrain Tokens (M)GRPO UpdatesDPO UpdatesTotal UpdatesAvg. SR (%) ↑
GRPO-only (Original)3,200076,800166.40200020077.62
GRPO-only (Budget-matched)3,712088,240191.65232023283.10
G-Harness (Full)3,45651289,088193.022161623289.39

Table 3. Local Optimization Under a Shared Worker Model

MethodWorker ModelHarness-Bench Score (%) ↑ALFWorld Avg. SR (%) ↑
Harness-R1GLM-5.3-Flash64.7283.22
HarnessXGLM-5.3-Flash74.4879.07
Meta-HarnessGLM-5.3-Flash72.4782.02
G-Harness (GRPO-only, Budget-matched)GLM-5.3-Flash69.8579.64
G-Harness (Global–Local)GLM-5.3-Flash77.8485.08

Q2: Are the structural and feedback ablations retrained under their modified configurations? Please specify which task information, observations, and completion signals remain available to the policy, workers, and runtime.

Response to Q2: Yes. The w/ Agent-Only Graph, w/o Task Graph, and w/o Runtime Feedback variants are independently retrained under their respective modified configurations, using separate SFT checkpoints and shared supervision sources with appropriately modified policy inputs. In contrast, w/o Online Rewriting reuses the trained checkpoint, constructs the initial Active Harness normally, and disables subsequent Policy calls.

Importantly, these ablations modify Policy inputs and decisions, rather than Worker observations or Runtime verification. Specifically, w/ Agent-Only Graph restricts the Policy representation to Agent nodes and Agent-to-Agent relations, with task allocation handled by the Runtime. w/o Task Graph removes explicit TaskNode goals, dependencies, capabilities, and completion contracts from the Policy encoder. w/o Runtime Feedback removes the explicit runtime observation oto_t, while retaining TaskState information (status, attempts, outputs, and errors), resource budgets, and legal actions. Workers retain their assigned task goals, completion contracts, predecessor outputs, and local environment observations, while the Runtime preserves the complete Task Graph and original completion-verification rules.

AuthorsOpenReview ↗

Response to Reviewer Nyo1 (Part 2 of 5)

Q3: Is the reported success rate averaged over individual tasks or equally across task categories? Please provide per-category success counts and denominators to clarify the aggregation.

Response to Q3: We thank the reviewer for requesting clarification on the evaluation metrics. For ALFWorld, the reported Avg. SR is computed as the macro-average across six task categories, rather than the task-level micro-average, i.e., Avg. SR=16∑c=16ScNc×100\mathrm{Avg.\ SR}=\frac{1}{6}\sum_{c=1}^{6}\frac{S_c}{N_c}\times100%, where ScS_c and NcN_c denote the successful and total task counts in category cc, respectively. The same aggregation protocol is applied to all methods, covering the complete 134-task valid-unseen evaluation set. Table 4 provides the per-category counts and denominators, distinguishing the macro-average (89.39%) from the micro-average (88.81%).

To further improve evaluation transparency, Table 5 reports per-task scores for all 21 Harness-Bench test tasks, while Table 6 presents a difficulty-level breakdown on DeepPlanning-Shopping (L1/L2/L3). Additionally, to assess run-to-run variability, we conduct multiple independent runs and report mean ± standard deviation for representative baselines and G-Harness in Table 7. G-Harness achieves 80.34 ± 0.87% on Harness-Bench and 88.10 ± 0.92% on ALFWorld, maintaining the highest mean performance among the compared methods. These additional analyses provide greater transparency regarding performance aggregation, task-level variation, and evaluation stability.

Table 4. Per-Category Success Rates on ALFWorld

Task CategorySuccessful / Total TasksSuccess Rate (%)
Look at Object in Light18/18100.00
Pick and Place21/2487.50
Pick, Clean and Place26/3183.87
Pick, Cool and Place21/21100.00
Pick, Heat and Place19/2382.61
Pick Two Objects and Place14/1782.35
Micro-average (Task-level)119/13488.81
Macro-average (Category-level)—89.39

Table 5. Per-Task Scores on Harness-Bench (Additional Evaluation Run)

TaskG-HarnessScore(%)
Task0161.34
Task0278.64
Task0381.92
Task0490.39
Task0583.77
Task0689.81
Task0773.64
Task0879.33
Task0990.71
Task1070.01
Task1180.69
Task1266.34
Task1384.18
Task1483.10
Task1585.27
Task1676.49
Task1792.84
Task1882.85
Task1975.54
Task2084.56
Task2175.72
Mean80.34

Table 6. Performance on DeepPlanning-Shopping

DifficultyTotal TasksAvg. Match Rate (%) ↑
L11092.80
L21087.60
L3472.68
Overall2487.28

Table 7. Performance Comparison and Stability Across Independent Runs

MethodHarness-BenchScore(%)↑ALFWorldAvg.SR(%)↑
Multi-AgentSystems
LEMON54.32±1.1462.25±1.21
AFlow71.23±0.8965.53±0.87
MAS-Orchestra62.07±1.2348.27±0.55
Harness-basedMethods
Harness-R165.62±0.9283.62±0.90
HarnessX75.38±1.3479.87±1.50
Meta-Harness71.97±0.7982.19±0.54
G-Harness80.34±0.8788.10±0.92

Q4: How can Table 5’s average consumption of 329.2K tokens be reconciled with the stated 256K per-task budget? Please clarify whether the execution budget, token-accounting convention, or maximum number of active agents changes in this experiment.

Response to Q4: We sincerely apologize for the lack of clarity in our original presentation and any confusion this may have caused. The 256K per-task token budget applies to the main benchmark comparisons to ensure fair evaluation. In contrast, the scalability analysis presented in Table 5 of the main paper increases the maximum number of active agents from 3 to 15 and removes the fixed token cap, allowing larger multi-agent configurations to execute without being artificially constrained by the original budget. Accordingly, the reported 329.2K tokens represents the actual average consumption of the 15-agent configuration under this separate setting.

More importantly, this experiment evaluates the scalability of G-Harness's graph-based orchestration mechanism. Rather than simply increasing the number of agents, G-Harness dynamically coordinates agent activation, task allocation, and interaction structures through its learned Graph Policy and runtime-aware graph rewriting, mitigating redundant or disorganized execution as the system scales. Notably, increasing the active-agent limit from 5 to 15 maintains nearly unchanged performance (80.68% → 80.90%), despite substantially higher token consumption (117.1K → 329.2K). These results suggest that G-Harness can maintain stable task performance under larger multi-agent configurations, while highlighting the diminishing returns and growing coordination costs of additional agents. We will clarify the distinct scalability setting and token accounting in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer Nyo1 (Part 3 of 5)

Q5: What evidence attributes the observed attack resilience specifically to local perturbation training or online rewriting, rather than model choice or deterministic validation? Evaluating the corresponding ablations under identical attack conditions would help isolate these contributions.

Response to Q5: We thank the reviewer for this important suggestion. To disentangle the contributions of our proposed mechanisms, we conduct additional robustness ablations under identical random-role agent attacks on ALFWorld, while keeping the worker-model configuration and deterministic validation rules unchanged. As shown in Table 8, the full G-Harness exhibits only a 1.14 pp decrease in success rate under attack (89.39% → 88.25%). In contrast, removing Online Rewriting increases the performance drop to 10.66 pp, while removing Local Graph Perturbation Optimization results in a 6.94 pp drop. These controlled comparisons indicate that both mechanisms contribute substantially to attack resilience.

Table 8: Robustness ablation under random-role agent attacks on ALFWorld. All variants are evaluated under identical attack conditions, with the same worker-model configuration, attack protocol, and deterministic validation rules. SR Drop denotes the absolute decrease in success rate under attack (percentage points).

MethodClean SR (%) ↑Attacked SR (%) ↑SR Drop (pp) ↓
w/o Online Rewriting79.4968.8310.66
w/o Local Graph Perturbation Optimization77.6270.686.94
G-Harness89.3988.251.14

Q6: How does G-Harness compare with strong single-agent systems equipped with comparable tools, memory, and reasoning budgets? Where applicable, comparisons with established agent harnesses such as Claude Code or Codex would help establish whether multi-agent orchestration provides benefits beyond stronger single-agent execution.

Response to Q6: We thank the reviewer for this valuable suggestion. To provide a stronger comparison beyond the seven vanilla single-agent baselines in our original experiments, we additionally evaluate two established agent harnesses, Claude Code and OpenAI Codex, on Harness-Bench, using the same evaluation protocol.

As shown in Table 9, G-Harness achieves 80.68%, outperforming Claude Code (73.60%) by 7.08 percentage points and OpenAI Codex (79.80%) by 0.88 percentage points. In terms of token consumption, G-Harness uses 117.1K tokens, compared with 135.2K for Claude Code and 93.1K for Codex. These results demonstrate that G-Harness remains competitive with established single-agent harnesses, achieving the highest task score among the compared systems.

Table 9. Comparison with Strong Single-Agent Harnesses on Harness-Bench.

MethodScore (%) ↑Tok. (K) ↓
Claude Code73.60135.2
OpenAI Codex79.8093.1
G-Harness80.68117.1

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.