- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MaRePID adds a bounded numerical controller to a frozen multi-agent LLM system. Process-evaluator scores and neighboring command states define an error; proportional, integral, and derivative terms produce an offset from an anchor command, which is clipped and converted into discrete prompt directives. The paper reports improvements on MultiAgentBench, MATH, BigCodeBench, and AgentWebBench, together with feedback-replay, PID-component, validation-budget, and optional sensing experiments. Its analysis derives command bounds, local affine dynamics, a scalar stability condition, and a tracking bound for the reference filter.
Critical strengths
-
The numerical interface makes the adaptation more explicit and inspectable. Separating evaluator measurements, calibrated error, bounded command state, and discrete prompt directives gives a concrete alternative to unconstrained textual revisions. The formulation accounts for dimensions, one-round measurement delay, integral clipping, and the fact that a numerical command change may leave the text unchanged.
-
The reported controls target relevant alternatives. Table 2 compares feedback regulation with replay, shuffled feedback, and a fixed dynamic schedule; Table 4 removes individual PID terms; Table 6 compares sensing against random skipping at matched query counts. These are useful tests of whether feedback timing and temporal memory contribute, and the reported results are promising if the experimental conditions can be established.
Critical weaknesses
-
The central empirical claims cannot be adequately assessed from the supplied experimental specification. Table 1 does not identify which of the three named backbones produces its scores or how results are aggregated. Test-set sizes, evaluator identity and rubric, reference-generation procedure, prompt templates, and the settings underlying the different aggregate scores in Tables 2–4 are not supplied. Appendix E largely refers back to Section 4, rather than providing these details; Appendix G names robustness conditions without reporting their outcomes. Although paired bootstrap intervals and Holm correction are promised, corresponding intervals or corrected test results are not shown. This matters especially for small differences such as 77.5 versus 77.4 in Table 3, and for reproducing the control interface itself.
-
The main text overstates what is established about closed-loop stability. Section 3.4 says Appendix C establishes stability of the nonlinear, quantized LLM system. What is shown is boundedness by clipping, conditional stability of a local affine model, and stability of a separate reference filter. Appendix C.3 explicitly identifies a unit eigenvalue in the unrestricted affine model; C.6 reports that the selected gains fail the scalar condition under its sensitivity substitution. These qualifications do not prove the deployed system unstable, but they mean the analysis does not establish its convergence or stable task behavior. The paper needs either a narrower claim or evidence that the relevant accessible dynamics satisfy the required conditions. Bounded prompts alone do not supply that evidence.
Questions for authors
- Please identify the backbone, evaluation sample counts, score aggregation, evaluator/model rubric, reference-generation and directive templates, and baseline inference budgets for each result table. What inputs can the evaluator and reference generator access, including any reference answers or tests? Are the promised per-backbone results, paired intervals, and robustness outcomes available? These details could substantially change confidence in the reported advantages.
- Which specific stability conclusion is intended for the deployed controller? Is there an identified accessible network model or an empirical tracking/convergence test that supports it, beyond clipping bounds and the conditional scalar analysis? Please reconcile the main-text claim with the qualifications in C.3 and C.6.
4 comments
Response to Reviewer K8pa (2/3)
Weaknesses (continued)
3. Stability claim.
Under stated assumptions, Appendix C proves three results. Commands and integral states stay in fixed boxes for any sensing and actuation decisions (Prop. C.6). The reference filter tracks its input with a geometric bound (Lemma C.7). A scalar local affine surrogate with inactive clipping is stable if and only if h(2K_P + K_I + 4K_D) < 2 (Thm. C.5). These results do not establish convergence or stability of the deployed nonlinear, quantized LLM loop. Section 3.4 now states exactly this and points to the tracking test in C.11. Section 5 is revised in the same way. Each run has six rounds and at most five command updates, so the method relies on bounded behaviour over a short horizon. Prop. C.6 guarantees this for the requested commands, not for the LLM's realized behaviour.
Questions
Q1a. Evaluator and its inputs.
The process evaluator is Qwen3-235B-A22B with thinking disabled, greedy decoding and a 768-token output cap. It returns task quality, constraint satisfaction and coordination in [0, 1]. Its system and user prompts are given verbatim in E.2. It receives six fields on every benchmark: TASK, ROLE, CONSTRAINTS, CURRENT_OUTPUT, PERMITTED_HISTORY and PUBLIC_OBSERVATIONS. None of them contains reference answers, hidden tests or other grading information, and BigCodeBench tests are used only for final scoring. Over 400 repeated scorings its standard deviation is 0.021 (C.3).
Q1b. Reference generation and directive templates.
The reference generator is a deterministic rule shared by all benchmarks. For each evaluator coordinate q, r_q = min(1, y_q^last + g_q) with g = (0.08, 0.04, 0.05). It makes no model call and sees no grading information. Eq. (3) then aligns references along the communication graph, and ζ_i clips the result to [0, 1] to give y_des. Section 3.2 now states this rule. E.3 gives the directive templates φ and the prompt composition Γ_i for every benchmark, and Table 8 now includes the AgentWebBench interface.
Q1c. Baseline inference budgets.
E.4 lists agent generations, evaluator calls, input and output tokens and wall-clock time per task for every method. In the MAIN family every multi-agent method makes 18 agent generations per task. MaRePID, Feedback-only and HiveMind also make 18 evaluator calls. MaRePID uses 189.8k tokens and 171 s per task, close to Feedback-only (188.8k, 173 s) and HiveMind (186.2k, 177 s). The offline optimizers GEPA, TextGrad and MIPROv2 make no test-time evaluator calls and use 124k to 132k tokens and 107 s to 109 s. Validation budgets are given in Table 3.
Q1d. Per-backbone results, paired intervals and robustness.
E.6 reports Table 1 for Qwen3-8B and DeepSeek-V4 with the same interface and gains. MaRePID improves over No-PID in every column on every backbone. Its AgentWebBench advantage over the strongest baseline is 4.8 points on Qwen3-8B and 4.6 on DeepSeek-V4, against 5.7 on Qwen3-235B-A22B. Across the 27 backbone-column cells, MaRePID is best or tied in 22 and below the strongest baseline in 5, by 0.2 to 1.7 points (largest: MATH on DeepSeek-V4). Paired intervals are in Weakness 2 above. Appendix G reports all robustness conditions against Feedback-only. The advantage holds under evaluator noise (+1.9 at SD 0.02, +0.6 at SD 0.05) and with two or five agents (+1.6 each). It nearly vanishes with a weaker evaluator (+0.2) and reverses under measurement delay (−0.3 for one round, −1.4 for two). Section 5 now states that the method relies on an accurate and timely evaluator.
Response to Reviewer K8pa (3/3)
Questions (continued)
Q2a. Intended stability conclusion.
For the deployed controller we claim only three things. Requested commands and integral states stay in fixed boxes for any LLM behaviour and any sensing or actuation decisions (Prop. C.6). The reference filter tracks its input with a geometric bound (Lemma C.7). A scalar local affine surrogate with inactive clipping is stable if and only if h(2K_P + K_I + 4K_D) < 2 (Thm. C.5). We do not claim convergence or stability of the deployed loop.
Q2b. Network model and empirical evidence.
We agree that bounded prompts alone do not show stable task behaviour. The revision adds an empirical test (C.11) on the logged CONTROL runs (160 tasks, 3 seeds, 3 agents, 1,440 agent runs, 28,800 coordinate updates). The mean tracking mismatch |y_des − y| falls from 0.0721 in round 1 to 0.0441 in round 6 (−39%), and the mean coordinate |e_i| falls from 0.0408 to 0.0250. No update reaches the offset saturation bound (0 of 28,800). Update directions reverse at least once in 64.4% of agent runs, which matches the mild oscillation the scalar model predicts for the most sensitive actuators. In 60.4% of coordinate updates the quantized directive level does not change. With five updates per run we present this as descriptive evidence, not as a convergence proof.
We have not identified the full local network model. That requires signed cross-coordinate and cross-agent sensitivities, while our finite-difference runs (C.5) perturbed one directive coordinate at a time. C.3 now states this limitation.
Q2c. Reconciling the main text with C.3 and C.6.
C.3. The unit eigenvalue belongs to directions v with Hv = 0. These are command changes the evaluator does not register, which exist because D_i has rank at most 3 < k = 4. Along these directions the error is zero, so they form a continuum of equilibria, not a growing mode. Residual inputs can still move the state along them without a restoring force, and the command box and integral clip of Prop. C.6 bound this movement. C.3 now says so explicitly.
C.6. You are right that with g = L_s at the 90th-percentile sensitivity, the selected non-root gains (0.50, 0.25, 0.08) violate the scalar condition in all four settings. The revised C.6 adds three facts.
- For non-root agents the condition is h < 2/1.57 ≈ 1.274, i.e. L < 1.457. The median sensitivities in Table 9 (0.87 to 1.21) satisfy it in all four groups, with spectral radius 0.84 to 0.89. In Research, Coding and Database the threshold lies above the 75th percentile, and in MATH it lies between the median and the 75th percentile.
- Root agents (gains 0.30, 0.09, 0.048) require h < 2/0.882 ≈ 2.27 and satisfy it even at the 90th percentile (at most 1.83).
- The substitution g = L_s is conservative in magnitude. L_s is the norm of the change across all three evaluator coordinates, while g uses one coordinate times its calibration coefficient (0.62 to 0.91).
The scalar surrogate therefore predicts local stability for typical actuators and possible oscillation for the most sensitive ones, most likely in MATH. We now present this as a limitation, not a guarantee. C.12 also checks the response sign the scalar model assumes. 18 of 20 directive coordinates respond in the intended direction, and the two exceptions in MATH are discussed in our response to Reviewer Fqec.
Thank you again for your review. We would be grateful to know whether our response and the revised PDF address your concerns, and we are happy to clarify anything further before the discussion period ends.
Response to Reviewer K8pa (1/3)
Thank you for a precise review and for separating what Appendix C proves from what Section 3.4 claimed. In short: (1) Appendix E now specifies every result table, the evaluator, the reference rule, inference costs and paired statistics. (2) After Holm correction, MaRePID is on par with the strongest baseline in every column of Table 1 except AgentWebBench, where it leads by 5.7 points. (3) The stability claim is limited to what Appendix C proves, and Appendix C.11 adds an empirical tracking test.
Weaknesses
1. Experimental specification.
All result tables use Qwen3-235B-A22B with seeds 17, 41 and 89, three agents, six rounds, full sensing and an update every round. Appendix E.1 assigns each table to an evaluation family.
ISOLATION and CONTROL reuse the 160 MAB-Coding tasks of Table 1 with a lenient score that omits the budget term. This raises scores by 2.4 to 4.9 points depending on the method, so MaRePID scores 74.6 in Table 1, 78.3 in Table 2 and 77.5 in Tables 3 to 5. ISOLATION and CONTROL were run separately and differ in the agents' maximum response length, which changes the accumulated history and token usage (175.2k vs 188.2k tokens per MaRePID task, Appendix E.4). We compare methods only within a family. Evaluation subsets are fixed by a hash-based rule (E.1). The first 60 IDs of each family form the validation set and are never used for testing.
2. Paired statistics.
Appendix E.5 reports a task-paired bootstrap (10,000 resamples, 95% percentile CI) with Holm correction over 11 comparisons against the strongest baseline in each column.
Only the AgentWebBench difference survives Holm correction, and the abstract and Section 4.2 now claim parity elsewhere. On AgentWebBench, MaRePID leads GEPA and Feedback-only in all four task families by 3.7 to 8.2 points, and all eight 95% CIs lie above zero.