- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces HICS (Heterogeneous Intelligence Coordination System), an external routing and quality-control framework for coordinating heterogeneous agents whose actual task-specific performance may differ from their public descriptions and whose feedback may be unreliable. The key idea is to learn each agent’s expected execution quality conditioned on the current task, using historical outcomes and feedback. HICS combines task-similarity-based local quality estimation with a direct contextual predictor to guide agent selection and exploration. It also separates answering from evaluation, maintains distinct estimates of executor and evaluator reliability, and uses reference feedback and labeled audit anchors to calibrate weak evaluations. Feedback is used to improve subsequent routing decisions rather than influence the decision that generated it.
Contributions
- Formulating multi-agent coordination around task-specific execution quality, distinguishing public task-fit descriptions from observed performance.
- Combining learned task geometry, local rolling quality estimates, direct contextual prediction, exploration, and separate executor/evaluator reliability modeling within HICS.
- Incorporating weak evaluations, reference feedback, and audit anchors to improve quality estimates when directly verified outcomes are limited.
Validation
- HICS with peer evaluation achieves 75.18% accuracy, outperforming NeuralUCB-Diag (71.80%) across four MedAgentBoard-based datasets.
- HICS achieves 73.17% full-stream and 80.92% final-window accuracy, outperforming LinUCB by 9.13 and 7.75 percentage points, respectively. However, full HICS uses additional evaluation requests, limiting the fairness of total-resource comparisons.
- Direct contextual prediction provides substantial gains, while online task-geometry learning has minimal impact in the tested simulations.
Critical strengths
- The paper addresses a meaningful problem by jointly modeling task-conditioned agent quality and evaluator reliability, recognizing that public agent descriptions may not reflect actual performance and that evaluation feedback can itself be unreliable. This provides a coherent framework for improving agent selection when directly verified outcomes are limited.
- HICS outperforms the strongest listed non-privileged baseline in both medical routing and controlled simulations. The ablation studies further identify direct contextual prediction as a major contributor to performance. These results provide empirical support for the framework, although the additional evaluation budget used by full HICS limits the strength of resource-matched comparisons.
Critical weaknesses
- Full HICS uses 2640 additional evaluation requests beyond the 7200 answer invocations shared by the compared methods. Consequently, its reported gains over LinUCB do not isolate the benefits of the proposed quality-learning framework from those of additional evaluation resources, weakening the evidence for its overall effectiveness.
- The framework relies on weak evaluations to update agent quality estimates, but the main simulations assume evaluators perform above chance and remain independent of answer-role collusion. Without stronger evaluation under coordinated or strategically manipulated evaluators, the paper does not sufficiently establish the reliability of its quality-control mechanism in the unreliable multi-agent settings it targets.
Questions for authors
- Can the authors quantify HICS's performance under a matched total interaction budget, including both answer and evaluation requests, to establish whether its gains over existing routing methods persist when resource consumption is controlled?
- How sensitive is HICS to evaluator unreliability, particularly when evaluators systematically provide incorrect feedback or collude with answering agents, and what evidence supports the effectiveness of its evaluator-reliability estimation under these conditions?
4 comments
2. Sensitivity to collusive evaluators (Weakness 2, Question 2)
Thank you for raising this concern. We have extended the collusive-evaluator analysis to examine how performance changes with the number of colluding nodes and the identity of the evaluator actually selected. The results are included in Appendix I.
The same colluding set can answer and evaluate. Its members retain correct outputs when successful and otherwise coordinate on a shared wrong answer; as evaluators, they favor that wrong answer and discourage the others. We increase the colluding population from 0 to 48 while keeping node abilities, effort, and potential correctness fixed.
Please see Figure 6 in Appendix I of the revised manuscript. It compares five methods at approximately 8% and 31% GT, with two curves for each method according to whether the selected evaluator is colluding.
Selecting a colluding evaluator is generally associated with lower accuracy, with a clearer separation as the colluding population grows. Performance also tends to decline at larger colluding populations. This deterioration is more pronounced with sparse GT: fewer verified outcomes leave the system more exposed to misleading feedback, whereas higher GT availability helps maintain accuracy. Across the plotted conditions, HICS achieves the highest mean accuracy in most cases, with a particularly consistent lead when a noncolluding evaluator is selected. LinUCB leads in a few of the remaining conditions.
The selection records also show that HICS limits exposure to colluding evaluators: with half the population colluding and sparse GT, only about 5% of its ordinary evaluation requests go to colluders. Together with the accuracy curves, this supports the value of learning evaluator reliability when feedback can be manipulated. The two curves describe outcomes under the system's actual selections and learning histories; evaluator judgments are used to update future decisions.
I appreciate the authors’ detailed rebuttal and substantial additional experiments. These new results partially address my original concerns, yet important limitations still exist. I will keep my rating as weak acceptance.
For the budget comparison. Results in Appendix G rule out that HICS gains purely come from extra evaluation calls. The augmented bandit baselines reuse HICS evaluator pipeline and do not natively model noisy weak feedback. It remains unclear whether a properly adapted bandit method could narrow the performance gap. A clear footnote should be added to main Table 3 to note its unbalanced‑budget setting and direct readers to equal‑budget comparisons in appendix.
For colluding evaluator robustness. The new collusion sweep shows the system can reduce calls to malicious evaluators. All tested adversaries follow fixed scripted behaviour. Adaptive strategic manipulation mentioned in the paper’s motivation is still unexamined. Relevant stress test data stays within appendices. The obvious performance degradation under heavy collusion needs explicit discussion in main limitations.
Thank you for taking the time to read our rebuttal and examine the additional experiments. We sincerely appreciate your careful follow-up and the specific suggestions for improving the paper.
We agree that our augmented bandit baselines retain their original learners while sharing a common evaluation pipeline, and that a dedicated treatment of noisy weak feedback could further improve their performance.
We have added a blue-marked budget note below Table 3, making the unequal budgets explicit and directing readers to Appendix G. We have also added a blue-marked paragraph in the main limitations discussing the degradation under heavy collusion, particularly with sparse GT, and clarifying that our adversaries follow fixed scripted behavior. Adaptive strategic manipulation remains unexamined.
The updated revision incorporates both changes. Thank you again for helping us make the comparison scope and the remaining limitations clearer.
1. Comparison under matched interaction budgets (Weakness 1, Question 1)
We apologize for the late response, following our recent CVPR submission. We greatly appreciate your careful review and your recognition of the framework's motivation. We agree that the original comparison combines the learning method with additional evaluation resources. We have therefore added request-matched comparisons with four baselines, using the original task streams, node pools, and GT masks. The complete HICS method is unchanged.
We compare the original baselines, HICS with evaluation removed, baselines augmented with evaluations, baselines given more answerers, and full HICS. The two augmented groups receive the same total number of requests as full HICS:
Accuracy (%), mean ± sample SD across three configurations. An asterisk (*) marks evaluation-augmented baselines; a dagger (†) marks answer-augmented baselines. All 2400 reports full-stream accuracy; Last 400 reports accuracy on the final 400 tasks.
The same results are in Table 11, Appendix G of the revised manuscript. Accuracy (%), mean ± sample SD over three configurations. An asterisk marks evaluation-augmented baselines; a dagger marks answer-augmented baselines. The evaluation budget comprises 2,400 ordinary and 240 anchor-pair requests. Evaluation-augmented methods retain their quality learners and use the same evaluator-selection, calibration, and weighting protocol. The added-answer group selects four nodes per task and a fifth on 240 predetermined tasks.
At 7,200 requests, HICS without evaluation exceeds the strongest baseline, LinUCB, by 2.78 points overall and 2.50 points in the final window. At 9,840 requests, full HICS exceeds evaluation-augmented LinUCB by 2.28/2.08 points and answer-augmented LinUCB by 3.51/2.75 points. The advantage thus remains when the extra budget is spent on either evaluations or answers. This comparison controls request count; token use, latency, and monetary cost are separate measures. The full results are now in Appendix G.