acceptodds
Desk rejected at ICLR 2027

RuralDoc: Evidence-Based Knowledge Graph Retrieval for Skin Disease Reasoning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Structured external knowledge is often introduced to improve large language model (LLM) reasoning, yet poorly integrated knowledge can instead override correct model beliefs. We study this failure mode in RuralDoc, a multimodal skin-disease reasoning framework that integrates retrieval-augmented generation (RAG) with a hierarchical diagnostic knowledge graph (KG). An initial ranking-based design presented KG predictions directly to the LLM and degraded performance for most backbones we tested, with deference analysis showing that models frequently abandoned correct initial answers when the KG ranking was wrong. We therefore redesign the KG as an evidence provider rather than a diagnostic authority: the revised method uses IDF-weighted matching over diagnostic symptoms and visual features, withholds disease rankings and scores from the LLM, and presents retrieved facts as unordered reference evidence alongside dermatology context. Across 192 diagnostic cases and five language models, the combined redesign removes the degradation observed with the original KG integration and improves Top-1 accuracy from 70.3% to 78.6% for Llama-3.1-8B and from 88.1% to 93.8% for DeepSeek-V3.1 relative to model-only baselines. In cases where the KG's top prediction is wrong but the model's initial answer is correct, erroneous deference drops sharply, from 67% to 7% for Llama-3.1-8B, with similar reductions for the other four models. These results show that the way structured knowledge is retrieved and exposed to an LLM can materially affect downstream reasoning and motivate treating KG outputs as supporting evidence rather than authoritative predictions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
knowledge graphs, large language models, retrieval-augmented generation, multimodal reasoning, medical AI, evidence retrieval

Related papers

Discussion (0)

Sign in to comment.