- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on dynamically modifying the configuration of multi-agent systems during the task process to utilize runtime conditions for better performance. The paper proposes G-Harness, trying to solve subtasks with different adaptive configurations (e.g., different agent roles and different model binding, etc.).
G-Harness treats a multi-agent system as a graph and dynamically selects active subgraphs to configure agent composition, task assignment, tool access, and model binding across subtask-level state transitions. A Global Harness Graph holds registered agents, tools, memories, candidate edges, and runtime state. An LLM Task Composer turns each task into a static directed acyclic graph. A task-conditioned policy then builds an Active Harness Subgraph and revises it during execution. Edits are drawn from a finite legal set: activate or deactivate components and edges, assign or unassign tasks, bind a model, set a configuration, request an additional registered capability, or commit the graph. The policy is basically a mapping from states to edit actions, with a specially designed encoding for graphs. Specifically, a relation-aware graph Transformer encodes over task, agent, tool, and memory nodes, with a frozen sentence-BERT encoder to produce description embeddings. The decoder is an MLP that maps the state encodings into graph-edit decisions (actions). The policy is trained by cloning trajectories from a teacher LLM, then by trajectory-level Group Relative Policy Optimization, and finally by a local preference stage that compares the committed graph with legal perturbations. After a graph is committed, execution continues until a task-state change or a material change in availability or resources opens another editing round.
The empirical case is made on Harness-Bench, DeepPlanning-Shopping, and ALFWorld. On the authors’ splits, G-Harness reports a Score of 80.68, a Match Score of 87.28, and an average success rate of 89.39 on the three benchmarks, respectively, above the vanilla, multi-agent, and harness baselines they evaluate, at a moderate reported token cost. An agent-level prompt attack on ALFWorld lowers its success rate by 1.14 points, from 89.39 to 88.25. Ablations show drops when the task graph, runtime feedback, online rewriting, GRPO, or local perturbation is removed, and two ALFWorld traces illustrate adding agents after insufficient exploration and rebinding a model after repeated illegal actions.
Critical strengths
The paper is generally well-structured, with a clear motivation and the corresponding solution to address the proposed scenario.
It basically gives insight similar to dynamic algorithm configuration (DAC): during a task process, a dynamic configuration is needed to deal with the different demands of the different stages or subtasks, and the runtime conditions and feedback can be utilized to do so. This insight has relatively extensive application scenarios in practical problems.
The dynamic graph modeling is interesting: selecting an active subgraph as the current configuration to support adaptation to different subtasks.
Experiments on three benchmarks support the excellent performance of the proposed method, as well as extra experiments on cost-performance trade-off and robustness tests.
Critical weaknesses
- Strictly speaking, the paper optimizes the structural configuration of the multi-agent system, with the tunable variables being the selection and connection on a pre-registered graph, rather than modifying the source code of the harness itself. Therefore, I think the statement "G-Harness, optimizing multi-agent harnesses" is not that precise. Especially when the policy of the proposed method needs to be retrained if there are new types of graph-edit operations. Moreover, the original Harness-Bench (the main benchmark in the paper's experiments) protocol binds one model and then compares different frozen harnesses, with no re-binding allowed mid-process, to fairly evaluate the pure capabilities of harnesses. However, the protocol of G-Harness is obviously different. It supports switching models midway, making it difficult to purely assess its capabilities as a "harness". This does not mean that the experiments cannot demonstrate the effectiveness of G-Harness. Instead, I think G-Harness is not suitable to be positioned as a method for improving harnesses, but is more suitable to be regarded as an approach for dynamically constructing MAS.
- Some formulas in the key formulation of the proposed graph transformer are not clearly described: line 226 ; line 232 are not explained.
- Many details supporting reproducibility are not specified: the size of the graph transformer, as well as many other hyperparameters in Local Graph Perturbation; the details of the "candidate scorer" (the paper just says it "assigns a scalar score", but what is it? an MLP?). Since the Task Graph is important in the method pipeline, the paper does not provide any examples or details of the Task Graph. In line 1071, the paper claims, "The released trace contains the Task Graph, the pre-edit and post-edit Active Harness snapshots, the accepted graph actions, the environment observations, and the completion evidence", but I cannot find anything about the Task Graph.
- The argument for the robustness test is a bit too sketchy. The paper only says "the robustness enabled by task-adaptive harness optimization and local graph perturbation training", but shows no traces that the strategy changed edges and switched models after detecting the lie. Moreover, the paper does not report which agent is attacked (only one of the five agents is attacked), nor whether the malicious output was adopted by other agents. For the single-agent baseline, replacing the system prompt amounts to attacking the sole decision-maker; for G-Harness, it is merely one of up to five roles. These two attack strengths are not equivalent.
- Fairness of experimental comparison. For the vanilla model baselines, the paper says, "Reasoning or thinking modes are disabled whenever supported", but does not explain the reason. The original conclusion of DeepPlanning is that the model with maximum reasoning strength enabled is significantly better than models from the same family with reasoning turned off. Moreover, the cost comparison is not quite fair, as only the token cost is reported. However, model sizes vary across different models, and their API prices differ as well. When it comes to costs, small models and large models have substantial gaps in price and latency for the same number of tokens. Some approaches may consume fewer tokens but rely entirely on large, expensive models with massive parameter counts. In that case, I believe the cost of such approaches is also very high.
Questions for authors
- Can you explain the reason why reasoning or thinking modes are disabled whenever they are supported for the vanilla model baselines?
- Can you provide more details that support reproducibility? For example, the details mentioned in Weaknesses.
- Can you provide more analyses of the robustness tests? For example, specifying which agent is attacked (especially the executor).
- Can you report cost from a fairer dimension? For example, the api cost?
- Can you provide more detailed examples of the specific tasks? For example, the Task Graph.
10 comments
Response to Reviewer xzE3 (Part 2 of 5)
W3: Many details supporting reproducibility are not specified: the size of the graph transformer, as well as many other hyperparameters in Local Graph Perturbation; the details of the "candidate scorer" (the paper just says it "assigns a scalar score", but what is it? an MLP?)… but I cannot find anything about the Task Graph. Q2: Can you provide more details that support reproducibility? For example, the details mentioned in Weaknesses. Q2: Can you provide more details that support reproducibility? For example, the details mentioned in Weaknesses. Q5: Can you provide more detailed examples of the specific tasks? For example, the Task Graph.
Response to W3, Q2 & Q5: We thank the reviewer for highlighting the need for more detailed implementation specifications and concrete Task Graph examples.
(1) Architecture and Training Details. Our original submission provides the Task Composer and training configurations in Appendix A.3 (Reproducibility Details), together with the graph-edit space and local perturbation procedure in Appendix B.1–B.2. We further provide the missing implementation settings, particularly the hyperparameters of Local Graph Perturbation Optimization, in Table 2. For further implementation details and experimental configurations, please refer to our anonymized code repository: https://anonymous.4open.science/r/ICLR-G-Harness-9CFF/. We will incorporate these additional settings into the camera-ready appendix.
(2) Candidate Scorer Implementation. The candidate scorer is a two-layer MLP (640 → 128 → 1) with GELU activation. It concatenates the 128-dimensional decoder state, the selected operation embedding, and up to three role-specific argument representations. Node arguments use Graph Transformer embeddings, while non-node arguments use projected text embeddings. The MLP assigns a scalar score to each Validator-approved argument tuple, and a softmax over legal candidates produces the conditional argument distribution.
Table 2. Additional Implementation and Training Configurations
| Category | Hyperparameter | Value |
|---|---|---|
| Graph Policy | Hidden Dimension | 128 |
| Graph Transformer Layers | 2 | |
| Attention Heads | 4 | |
| Dropout | 0.0 | |
| Reward Normalization | Execution-cost Component Weights | (1.0, 1.0, 1.0, 1.0, 1.0) |
| Execution-cost Normalization Scales | (96K tokens, 50 calls, 600s, 1 CNY, 4 expansions) | |
| Graph-cost Component Weights | (1.0, 0.2, 0.5) | |
| Graph-cost Normalization Scales | (16 nodes, 32 edges, 32 edits) | |
| Graph Editing | Maximum Graph-edit Budget | 32 |
| Local Graph Perturbation | DPO Scale | 0.1 |
| Local-loss Weight | 0.1 | |
| Maximum Perturbed Alternatives per Branch Point | 2 | |
| Maximum Edits per Perturbation | 3 | |
| Perturbation Search Frontier Limit | 64 | |
| Maximum Refinement Updates (Harness-Bench / ALFWorld) | 32 / 125 |
(3) Task Graph Construction and Examples. We thank the reviewer for requesting concrete Task Graph examples. The Task Graph is formally defined in Section 3 (Task Graph), with its construction and validation procedures detailed in Appendix A.3 (Reproducibility Details). To further clarify its construction, we provide representative examples from both ALFWorld and Harness-Bench in the examples/ directory of our anonymized code repository (https://anonymous.4open.science/r/ICLR-G-Harness-9CFF/). Specifically, each example includes a machine-readable Task Graph (task_graph.json) and its visualization (task_graph.svg), together with Active Harness snapshots, graph-edit decisions, and execution traces. These examples illustrate how task-specific dependencies are represented and how the harness adapts during execution.
Response to Reviewer xzE3 (Part 3 of 5)
W4: The argument for the robustness test is a bit too sketchy. The paper only says "the robustness enabled by task-adaptive harness optimization and local graph perturbation training", but shows no… Q3: Can you provide more analyses of the robustness tests? For example, specifying which agent is attacked (especially the executor).
Response to W4 & Q3: We thank the reviewer for highlighting the need for a more detailed robustness analysis, particularly regarding attack-target specification, evaluation fairness, and concrete evidence of runtime graph adaptation. We address these concerns through additional controlled comparisons, robustness ablations, and an Executor-targeted execution trace.
(1) Attack Protocol and Executor-Targeted Evaluation. Following G-Designer[1], our original robustness evaluation applies a system-prompt attack to one eligible task-performing agent, selected deterministically using the task identifier and random seed 42. To address the concern that compromising the sole decision-maker in a single-agent system differs from attacking an arbitrary role in a multi-agent system, we additionally conduct Executor-targeted attacks under a shared GLM-5.3-Flash worker configuration (Table 3). Specifically, we target the sole Executor in Vanilla, the task-executing agent in LEMON, and the Executor in G-Harness. Under this setting, Vanilla and LEMON suffer success-rate drops of 13.63 and 10.60 percentage points, respectively, whereas G-Harness exhibits a smaller drop of 6.01 points, retaining 79.27% attacked SR. These results provide a more direct assessment of robustness when the critical execution role is compromised, rather than relying solely on random-role attacks.
(2) Robustness Mechanism Ablation. To investigate whether robustness arises from the proposed mechanisms rather than multi-agent redundancy alone, we further evaluate two ablations under the original random-role attack protocol (Table 4). Removing Online Rewriting reduces attacked SR from 88.25% to 68.83%, with the SR drop increasing from 1.14 to 10.66 points. Removing Local Graph Perturbation Optimization yields 70.68% attacked SR and a 6.94-point degradation. These results support the contributions of both task-adaptive runtime graph rewriting and local graph perturbation optimization to adversarial robustness.
(3) Attack Propagation and Runtime Graph Reconfiguration. To provide concrete evidence of runtime adaptation under adversarial attacks, we include an Executor-targeted ALFWorld case study in examples/Example_executor_attack.json of our anonymized code repository (https://anonymous.4open.science/r/ICLR-G-Harness-9CFF/). In this example, the attacked Executor produces misleading information that is received but not adopted by the Reasoner. Based on contradictory environment feedback, G-Harness deactivates the affected communication edge, reassigns the localization task, and switches the Reasoner's model from GLM-5.3-Flash to Qwen3.7-Plus, ultimately completing the task successfully. The trace documents the attack propagation and corresponding graph-edit actions.
Table 3. Robustness Under Random and Executor-Targeted Agent Attacks on ALFWorld
| Attack Setting | Method | Worker Model | Attacked Role | Clean SR (%) ↑ | Attacked SR (%) ↑ | SR Drop (pp) ↓ |
|---|---|---|---|---|---|---|
| Original Random-role Attack | G-Harness | Original Model Pool | Random Eligible Agent | 89.39 | 88.25 | 1.14 |
| Shared-worker Executor Attack | Vanilla Agent | GLM-5.3-Flash | Sole Executor | 58.96 | 45.33 | 13.63 |
| LEMON | GLM-5.3-Flash | Task-executing Agent | 61.84 | 51.24 | 10.60 | |
| G-Harness | GLM-5.3-Flash | Executor | 85.28 | 79.27 | 6.01 |
Table 4. Robustness Ablation Under Random-role Agent Attacks on ALFWorld.
| Method | Clean SR (%) ↑ | Attacked SR (%) ↑ | SR Drop (pp) ↓ |
|---|---|---|---|
| w/o Online Rewriting | 79.49 | 68.83 | 10.66 |
| w/o Local Graph Perturbation Optimization | 77.62 | 70.68 | 6.94 |
| G-Harness | 89.39 | 88.25 | 1.14 |
Response to Reviewer xzE3 (Part 1 of 5)
Dear Reviewer xzE3,
Thank you for your thoughtful review and constructive feedback. We appreciate your recognition of the motivation, dynamic graph formulation, and empirical effectiveness of G-Harness. We address these concerns and provide additional clarifications and experimental evidence below.
Response to W1: We thank the reviewer for pointing out this important distinction and for the opportunity to clarify the scope of harness optimization in G-Harness and its relationship to the original Harness-Bench evaluation protocol.
(1) Scope of Harness Optimization. Consistent with existing harness research (e.g., Harness-Bench[1] and HarnessX[2] ), we view a multi-agent harness as the runtime execution and control layer governing the coordination of agents, tools, memory, and model interactions. Accordingly, harness optimization is not limited to source-code modification, but also encompasses the optimization of runtime configurations and control decisions. G-Harness specifically optimizes component activation, interaction topology, task assignments, tool access, and model bindings based on task requirements and execution feedback. Thus, dynamic MAS construction is an integral part of G-Harness's runtime harness optimization, which further extends to continuous structural reconfiguration and learning-based graph-edit policy optimization.
(2) Extensibility of Graph Editing. We acknowledge that introducing fundamentally new graph-edit operation types may require extending and retraining the policy. However, this does not limit G-Harness to predefined harness configurations. While the graph-edit operation types are fixed, the learned policy dynamically determines which components and interactions to modify, when to perform these edits, and how to combine them based on task requirements and runtime feedback. Moreover, existing operations can be applied to newly registered compatible components without changing the action vocabulary.
(3) Benchmark Protocol and Model Rebinding. The original Harness-Bench protocol focuses on evaluating fixed model–harness configurations, while G-Harness investigates runtime-adaptive harness optimization. In our setting, dynamic model rebinding is an integral harness-level control mechanism, allowing the system to adapt when a worker model becomes insufficient for evolving task requirements. To further disentangle the benefits of structural adaptation from heterogeneous model capabilities, we additionally evaluate G-Harness using GLM-5.3-Flash exclusively, without cross-model switching (Table 1). Under this setting, G-Harness achieves 77.84% on Harness-Bench and 85.08% on ALFWorld, outperforming the strongest fixed-worker baselines by 3.36 and 1.86 percentage points, respectively. These results demonstrate that G-Harness's performance advantages are not solely attributable to model switching, but also arise from adaptive harness configuration and structural optimization.
Table 1. Local Optimization Under a Shared Worker Model
Response to W2: We thank the reviewer for identifying these missing definitions. We clarify the four parameters as follows:
These parameters enable type-aware node encoding and relation-aware attention, allowing the Graph Transformer to distinguish heterogeneous components and their interactions. We will explicitly define these parameters in the camera-ready version.