- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RareVA, a multi-agent framework for identifying rare diseases from facial images. RareVA uses a visual classifier to select top-k candidates and a VLM to extract phenotypes. Then, doctor agents verify selected candidates and a judge agent outputs the final prediction. The adaptive candidate Selection (ACS) policy is proposed to allocate token budgets. Experiments show that RareVA can improve accuracy while reduce token costs.
Critical strengths
- The motivation is clear and easy to follow. The candidate-conditioned verification formulation provides a clearer and more explainable way to identify rare diseases.
- Connecting knowledge bases like HPO with the image is practical.
- The experimental section of RareVA provides detailed statistical metrics like confidence intervals, making the reported results more reliable.
Critical weaknesses
- RareVA first uses a vision classifier to output the candidate diseases and a VLM extracts phenotypic evidence. Why don't let the VLM also output the candidate diseases?
- Some experimental settings are not clear, making the method difficult to reproduce. What is the trainable part in RareVA? Is it only the ACS policy and vision classifier? What is the detailed training settings of ACS?
- In the ablation of Table5, the performance of 'w/o ACS K=4' is higher than RareVA, and is statistically better. This makes the ACS less valuable if its function is just to save tokens. Since using a multi-agent system is inherently token-intensive, performance improvement is more important. And why isn't the bolded result the best one?
- Setting the max tokens as 1k limits the capability of the LLM.
Questions for authors
- What does the '¬' mean in Line 7 of Algorithm 1?
6 comments
Response to Reviewer p3QS — Part 2/2: W3, W4, Q1 — ACS Effectiveness and Accuracy–Cost Trade-off, Generation Length, and Algorithm Notation
W3. ACS Effectiveness and Accuracy–Cost Trade-off
We thank the reviewer for raising this important concern regarding ACS effectiveness and the importance of predictive accuracy.
(1) We clarify the statistical comparison. Fixed K=4 does not significantly outperform ACS on RDFace (McNemar's exact p=1.0; paired bootstrap 95% CI for ACS − K=4: [−6.19, 4.42] pp; Section 4.7; Appendix K.3). The observed Top-1 accuracies are 35.40% and 34.51%, respectively.
(2) We distinguish ACS from the diagnostic verification framework. Without ACS, K becomes a globally selected hyperparameter; the candidate-conditioned Doctor–Judge framework remains unchanged. ACS instead learns case-dependent verification allocation from downstream utility (Section 3.3.1; Appendix E.2–E.4). Thus, the diagnostic accuracy contribution comes primarily from Doctor–Judge verification, while ACS addresses inference allocation under computational constraints (Sections 3.3.2–3.3.3; Section 4.8, Table 5).
Table R15. Accuracy–cost comparison on RDFace.
| Verification policy | Top-1 Acc. (%) | Avg. output tokens |
|---|---|---|
| Fixed K=3 | 33.63 | 7,834 |
| Fixed K=4 | 35.40 | 10,064 |
| Entropy-based routing | 29.20 | 7,154 |
| ACS (Ours) | 34.51 | 8,338 |
| Results from Section 4.8 (Table 5) and Appendix E.3 (Table E.1). |
(3) We additionally evaluated ACS on the independent GMDB cohort (n=661) to examine its performance beyond RDFace. The RDFace-trained ACS policy was transferred without target-domain retraining, and all configurations used the same candidate-generation and Doctor–Judge verification protocol.
Table R16. External evaluation of verification budgets on GMDB.
| Verification policy | Top-1 Acc. (%) | Avg. output tokens |
|---|---|---|
| Fixed K=3 | 11.65 | 7,674 |
| Fixed K=4 | 11.95 | 10,178 |
| ACS (avg. K=3.21) | 12.25 | 8,198 |
Importantly, ACS achieves the highest observed accuracy on GMDB while reducing inference cost relative to fixed K=4. It improves Top-1 accuracy from 11.95% to 12.25% while using 19.45% fewer output tokens. Together, the RDFace and GMDB results indicate that adaptive verification does not necessarily require sacrificing predictive accuracy for efficiency, and provide additional evidence for transferring learned verification allocation across cohorts.
Finally, we have also corrected the bold formatting in Table 5 to highlight fixed K=4 (35.40%) as the highest-accuracy configuration on RDFace in the revised PDF.
W4. Sensitivity to Maximum Generation Length
We thank the reviewer for raising the concern that the 1,024-token generation limit may constrain the reasoning capability of the VLM agents. To directly assess this, we conducted an additional sensitivity analysis on RDFace by varying the maximum output tokens per generation from 800 to 4,096 (4× the original limit). We use fixed K=4 to isolate the effect of generation length from adaptive budget allocation, keeping all other inference settings unchanged (Appendix M, Tables M.2–M.3).
Table R17 (corrected). Sensitivity to maximum output tokens per generation on RDFace (fixed K=4).
| Max output tokens / call | Top-1 Acc. (%) | Correct / 113 |
|---|---|---|
| 800 | 32.74 | 37 |
| 1,024 (original) | 35.40 | 40 |
| 2,048 | 34.51 | 39 |
| 4,096 | 34.51 | 39 |
| Incomplete or invalid agent generations are counted as incorrect rather than excluded. |
Increasing the generation limit beyond 1,024 does not improve observed Top-1 accuracy: both 2,048 and 4,096 tokens yield 34.51% (39/113), compared with 35.40% (40/113) at the original limit. Conversely, reducing the limit to 800 tokens lowers accuracy to 32.74% (37/113).
These results indicate that the 1,024-token cap is not an observed accuracy bottleneck within the evaluated range, while avoiding the claim that it is globally optimal. We have included this sensitivity analysis into the revised manuscript (Appendix M, Tables M.2–M.3).
Q1. Notation in Algorithm 1
The symbol in Line 7 denotes logical negation (NOT), indicating that the Doctor Agent's stopping criterion has not yet been satisfied. Thus, the verification loop continues until either the maximum number of rounds is reached or the stopping condition becomes true. We also noticed an extra comma (,) in this line, which is a typo error. We have removed it and clarified the condition as while round < T_max and not STOP(L_Doctor) do in the revised manuscript.
Thanks for the responses. The rebuttal addressed some of my concerns, but the responses to W1 and W4 are not convincing. The VisCls is trained, but the VLM is not trained, so it is not a fair comparison. And the results in Table R16 (by the way, it should be R17) are statistically unreliable with only 113 samples. I maintain my score.
Response to Reviewer p3QS — Part 1/2: Summary and W1, W2 — Candidate Generation, Trainable Components, and ACS Training Reproducibility
Dear Reviewer p3QS,
Thank you for your constructive feedback and for recognizing the motivation, interpretability, and statistical rigor of RareVA. To address your concerns, we conducted additional experiments on VLM-based candidate generation (W1, Table R12), ACS accuracy–efficiency trade-offs on RDFace and GMDB (W3, Tables R15–R16), and generation-length sensitivity (W4, Table R17). We also provide detailed clarification of RareVA's trainable components and ACS training protocol (W2, Tables R13–R14).
Notably, our results show that the task-specific visual classifier achieves substantially higher Top-5 candidate recall than direct VLM-based proposals (43.36% vs. 8.85–10.62%). Furthermore, ACS maintains competitive accuracy while reducing output-token usage by 17.15% on RDFace and 19.45% on GMDB relative to fixed K=4. Increasing the generation limit beyond 1,024 tokens also does not improve observed accuracy.
We have updated the revised manuscript to incorporate these additional analyses, implementation details, and clarifications. We hope these results help address your concerns. Our detailed, point-by-point responses are organized into two parts, beginning with W1 below.
W1. Why Not Use the VLM for Candidate Generation?
RareVA is designed to separate visual candidate generation from phenotype-grounded VLM verification (Section 3.1). The task-specific visual classifier proposes a bounded set of disease hypotheses from facial images, allowing VLM agents to focus on phenotype interpretation, KG-grounded evidence assessment, and candidate-conditioned reasoning (Section 3.3). This separation is a deliberate architectural choice rather than an assumption that VLMs cannot generate disease candidates.
To empirically assess the reviewer's suggestion, we conducted an additional comparison using the same VLM backbone (Gemini) as RareVA's reasoning agents: (1) direct VLM proposal, given the facial image and all 103 target disease labels; and (2) VLM with in-context learning (ICL), additionally given one labeled training image per disease. Both generate Top-5 candidates on the same RDFace test set.
Table R12. Top-5 ground-truth candidate recall on RDFace.
The additional results provide empirical support for our original architectural choice. As in Table R12, VisCls achieves substantially higher Top-5 recall (43.36%) than direct VLM proposal (8.85%) or VLM with ICL (10.62%) under the evaluated protocol. Thus, RareVA retains a task-specific visual candidate generator while using VLMs for structured verification rather than unrestricted disease proposal (Section 3.1; Appendix C.1).
W2. Trainable Components and ACS Training Reproducibility
We thank the reviewer for highlighting the need for clearer implementation details. Yes, only the visual classifier (VisCls) and ACS policy are trained specifically for RareVA. All VLM components remain frozen and operate through prompting, while phenotype normalization and KG retrieval require no additional training (Section 3; Appendix M). In Table R13, we summarize the details of each component in RareVA.
Table R13. Trainable components in RareVA.
ACS is trained through offline utility regression (Appendix E.2–E.3) using the 46-image RDFace validation split, with downstream rollouts for each K∈{2,3,4,5}. The utility balances prediction correctness and output-token cost (Appendix E.3, Eq. 12):
$
u(x,K)=1[y^K=yGT]−λcost(x,K).$
An MLP predicts utility from pre-reasoning routing features and a one-hot encoding of K, selecting the highest-utility budget at inference. Hyperparameters are selected via 5-fold cross-validation on the validation split, followed by training a single policy checkpoint. The test set is excluded from training and model selection, and inference uses neither ground-truth labels nor downstream reasoning outcomes. The training configuration is summarized in Table R14.
Table R14. ACS training configuration (Appendix M, Table M.4).
We have incorporated these implementation details into Appendix M (Tables M.4) to improve reproducibility.