acceptodds
Desk rejected at ICLR 2027

RSIMed: Scaling Medical Foundation Models beyond Limited Clinical Data via Retrieval-Grounded Self-Evolution

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Recursive Self-Improvement (RSI) has gained wide attention for its ability to work with limited data. In a proposer-solver loop, a proposer writes training tasks and a solver learns to solve them through reinforcement learning. Such loops succeed in mathematics and coding, where a program checks every answer. Medicine, however, has scarce, private records and no such verifier, so a wrong model-written reference answer can corrupt training. To solve this problem, we introduce RSIMed, Recursive Self-Improvement in Medicine, one of the first self-evolving pipelines where the proposer co-evolves with the solver in context, on free-text diagnosis from real clinical records. The proposer, a frozen copy of the solver, writes every task from a retrieved medical database and adjusts its difficulty and topics to the solver's live accuracy and past tasks. On rare-disease diagnosis from MIMIC-IV, the accuracy of RSIMed-27B grows linearly with training compute, while a static pool of tasks written once collapses. Without any external model, RSIMed-27B and RSIMed-9B reach 0.516 and 0.471 accuracy, above every GPT-5.6 configuration, whose best reaches 0.399 at 38 times the inference cost of RSIMed-27B. In particular, we see that a stronger teacher does not always produce a stronger student. We further show that the learned skills generalize, as RSIMed-RL-27B raises its adjusted score on out-of-distribution HealthBench Professional from 0.291 to 0.303.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Medical foundation models, self-evolution, synthetic data, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_n57UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RSIMed, a retrieval-grounded self-improvement framework for medical foundation models under limited labeled clinical data. The framework combines a frozen task proposer, evidence retrieval, task validation, and reinforcement learning. The proposer generates medical questions from retrieved references and adapts their coverage and difficulty using feedback from the solver. The solver is trained through supervised fine-tuning followed by reinforcement learning with semantic rewards.

Critical strengths
  1. This article addresses the problem that clinical annotation data is limited and the generated answers cannot be verified through executable tests.
  2. Splitting the solver and task proposer is an intuitive and efficient solution.
  3. RSIMed achieves substantial improvements on rare-disease diagnosis, including gains over supervised initialization and static synthetic-data training.
Critical weaknesses
  1. The ablation experiment failed to truly reflect the proposed innovation. The main comparison is against reinforcement learning on labeled cases or a fixed synthetic-task pool. Continuous generation changes several factors simultaneously, including the number, diversity, and difficulty of available tasks. In particular, the static control uses 15,000 generated tasks, whereas the continuous run analysis 43,698 generated task instances served during training.
  2. The validator rejects answers directly contradicted by retrieved evidence but accepts tasks when the evidence is inconclusive. It remains unclear how incorrect reference answers affect reinforcement learning or whether the validator consistently controls their accumulation.
  3. The results in out-of-domain scenarios undermine the conclusion that the full framework consistently enhances general clinical capabilities. In the HealthBench Professional benchmark, a variant utilizing only RL improved the score from 0.291 to 0.303, whereas SFT+RL achieved a score of only 0.280.
  4. Figure 2 would be more informative if it could be populated with actual content.
Questions for authors

Please see the weakness.

2 comments
AuthorsOpenReview ↗

Thanks for the review and response to questions

We thank the reviewer for the review. We address each point below.

The ablation experiment failed to truly reflect the proposed innovation. … Continuous generation changes several factors simultaneously, including the number, diversity, and difficulty of available tasks. In particular, the static control uses 15,000 generated tasks, whereas the continuous run analysis 43,698 generated task instances served during training.

We ran the requested experiments. We see that with the number of fresh tasks matched, feedback still makes the solver learn faster. The results below are done with the RSIMed-9B recipe with no feedback and matched sample sizes. We use the same Qwen3.5-9B proposer/judge at training time, and GPT-5.5-Chat judge for validation scoring.

RunAccuracyLearnable tasks
RSIMed-9B0.42619.8%
No feedback0.41311.0%

Note the accuracy here is the mean from step 200 to 440, which is far from the saturation performance. We see that even at matched sample size, RSIMed-9B is still the best model with highest learning signal. We further see that the feedback allows the proposer to propose a higher proportion of useful tasks, at 19.8% vs 11.0% for no feedback variant.

We would like to further note that the static pool variant fell from its 0.441 peak to 0.359 by step 155, after only 9,920 draws from its 15,000 tasks. Therefore, repeated training is not a cause for its collapse, and even if it has 43k tasks, it will still collapse well before all tasks are consumed.

The validator rejects answers directly contradicted by retrieved evidence but accepts tasks when the evidence is inconclusive. It remains unclear how incorrect reference answers affect reinforcement learning or whether the validator consistently controls their accumulation.

We see that wrong keys stay rare and carry little learning signal. We audited a random sample of 100 accepted short-answer tasks from each fifth of training in RSIMed-27B and RSIMed-9B with GPT-5.5-Chat as automatic judge. We see that:

  • They do not accumulate. The valid share rises from the first fifth of training to the last in both runs: from 78.2% to 86.5% in RSIMed-27B and from 72.0% to 80.0% in RSIMed-9B.

  • Invalid tasks usually do not get any learning signal. 73% of invalid tasks get no rollout that matches the key, against 54% of valid tasks, so they give no correctness signal.

  • We further replayed a strict version of the validator, which accepts a task only when the retrieved evidence supports its key, with a frozen Qwen3.5-9B checker. The strict version raises the valid share from 81% to 88% but rejects 19% of the valid tasks. As a comparison, the paper's validator keeps 98% of the valid tasks and rejects 11% of the invalid tasks. In other words, requiring support removes 2.3 units of valid signal for every unit of invalid signal.

The results in out-of-domain scenarios undermine the conclusion that the full framework consistently enhances general clinical capabilities. …

We would like to point out that the main focus of the work, and all improvements on the self improving loop, is on the RL part of the pipeline. Therefore, RL only result being better actually indicates that the self improving pipeline gives a more generalizable OOD performance that normal SFT on generated data does not provide, which strengthens our claim. If we break down the results by category, we see that RL only is at or above the untrained model in every category with zero targeted training data for HealthBench Pro:

Good faithRed teamingTypicalDifficultConsultationResearchWriting
+0.010+0.017+0.001+0.023+0.016+0.001+0.017

With that said, we will further adjust the claim to make this more clear.

Figure 2 would be more informative if it could be populated with actual content.

We will fill Figure 2 with the real task of Figure 3, from seed case to graded solver responses.

We will further make the minor fixes mentioned in the review, and consolidate the new results above into the main text.

AuthorsOpenReview ↗

Thanks for the review

Dear Reviewer n57U,

Thanks for spending the time to write the review. As the discussion period ends today, we hope our response addresses your concerns. We are happy to take any follow up clarifying questions you may have.

Reviewer_qy7xOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RSIMed, a retrieval-grounded training pipeline for clinical diagnosis. A frozen proposer generates medical tasks and adapts its prompts using solver history and accuracy, while a solver learns through reinforcement learning. Generated tasks are checked against retrieved evidence, and free-text answers receive a composite reward. The paper reports substantial gains on MIMIC-Rare, comparisons with fixed-data training and external task generators, and a small HealthBench Professional improvement from an RL-only variant.

Critical strengths
  1. The paper addresses a relevant practical problem that extending medical training beyond limited labeled cases while handling free-text clinical answers.
  2. The empirical contribution is useful. Both model sizes show substantial diagnostic gains, and the full larger-model pipeline improves over supervised initialization and both fixed-data RL controls.
  3. The paper includes useful task-quality audits, task-source analyses, and training-stage comparisons. It also reports the full pipeline’s weaker out-of-distribution performance.
Critical weaknesses
  1. The benefit of adaptive feedback is not clearly separated from that of continued task generation. The fixed-pool comparison changes both factors, so the results support the complete pipeline but do not establish how much solver-conditioned generation contributes. The authors should provide a matched continuous-generation comparison without solver feedback or narrow the corresponding mechanism claim.
  2. The limited-data setting and main diagnostic task need clearer documentation. The paper does not clearly report the number of labeled training cases, whether the split is by patient or admission, or how the target diagnosis and clinical input are constructed. The authors should provide these details so that the setting and reported gains can be assessed and reproduced.
  3. The methodological novelty is incremental. The main advance is adapting and integrating established methods for clinical free-text tasks with a frozen proposer. The authors should clarify the new design choices and empirical findings relative to prior work.
Questions for authors

See critical weaknesses

3 comments
AuthorsOpenReview ↗

Thanks for the review and response to questions - Part 1

We thank the reviewer for the positive assessment and the precise requests. We ran the requested matched comparison and document the dataset below.

The benefit of adaptive feedback is not clearly separated from that of continued task generation. … The authors should provide a matched continuous-generation comparison without solver feedback or narrow the corresponding mechanism claim.

We ran the requested experiments. We see that with the number of fresh tasks matched, feedback still makes the solver learn faster. The results below are done with the RSIMed-9B recipe with no feedback and matched sample sizes. We use the same Qwen-3.5-9B proposer/judge at training time, and GPT-5.5-Chat judge for validation scoring.

RunAccuracyPercentage of task with Learning Signal
RSIMed-9B0.42619.8%
No feedback0.41311.0%

Note the accuracy here is the matched accuracy at no feedback's best step, which is far from the saturation performance. We see that even at matched sample size, RSIMed-9B is still the best model with highest learning signal. We further see that the feedback allows the proposer to propose a higher proportion of useful tasks, at 19.8% vs 11.0% for no feedback variant.

We would like to further note that the static pool variant fell from its 0.441 peak to 0.359 by step 155, after only 9,920 draws from its 15,000 tasks. Therefore, repeated training is not a cause for its collapse, and even if it has 43k tasks, it will still collapse well before all tasks are consumed.

The limited-data setting and main diagnostic task need clearer documentation. The paper does not clearly report the number of labeled training cases, whether the split is by patient or admission, or how the target diagnosis and clinical input are constructed.

We explain each of the requested information below:

On cases, we use 8,171 MIMIC-IV admissions, split by both date and patients for the greatest separability. That is, we use all admissions from the first 4,093 patients chronologically apprearing in the MIMIC-IV dataset for training, and the rest for evaluation. Together, training has 5,719 admissions from 4,093 patients, and evaluation has 2,452 admissions from 1,713 other patients.

The label for each case is the admission's primary ICD-10 diagnosis, kept only if the code or its prefix maps exactly to an Orphanet rare disease. Training covers 375 codes and evaluation 269. Out of all evaluation labels, 52 evaluation cases (2.1%) have a code unseen in training.

The input is the structured record (demographics, labs, vital signs, microbiology, procedures, medications), radiology reports, up to two chest X-rays and one ECG. We exclude the target admission's discharge summary and retain only notes from prior admissions as patient history. Concretely, any note whose chart time falls at or after the target admission's admit time is removed, so that no diagnosis-revealing or outcome-encoding text from the current stay enters the input. The text is capped at 9,000 characters.

We will add these details to the Datasets paragraph and the appendix.

AuthorsOpenReview ↗

Thanks for the review and response to questions - Part 2

The methodological novelty is incremental. … The authors should clarify the new design choices and empirical findings relative to prior work.

We would like to note that RSIMed is one of the first self-evolving pipelines where the proposer co-evolves with the solver in context, on free-text diagnosis from real clinical records. You may see how it compares against prior works in related works section and Table 2. Specifically, we:

  1. Provide the proposer with step by step dynamic feedback so it co-evolves with the solver in context. Absolute Zero, R-Zero, SPICE and WIST train the proposer with RL to write tasks at the edge of the solver's ability [1–4]. This second training loop needs a reward for good tasks. In medicine, this would need a correctness check that does not exist. Prior literature also show a trained proposer also learns to hack that reward until self-play stalls [5]. RSIMed keeps the proposer frozen and feeds the solver's recent results into its context. It has no reward to hack, yet it follows the solver as it improves, which writes 1.8 times as many learnable tasks as the same proposer without feedback, from the table above.

  2. Our validator keeps the generated keys correct. Prior loops check answers by code execution [1], majority vote [2], rule matching [3, 4] or a proof compiler [5], and none of these works for a free-text diagnosis. RSIMed writes every reference answer from retrieved medical literature, re-checks it against freshly retrieved passages, and grades diagnoses for meaning. Without such a check, wrong generated labels reinforce the model's own errors [6].

  3. We establish a scaling law where we see Accuracy grows linearly with training compute. RSIMed-27B rises from 0.421 to 0.516 on held-out patients, 3.9 points per 102010^{20} FLOPs, with no plateau in the observed range. A straight line fits all 192 evaluations with R2=0.96R^2 = 0.96. From the same warm start, RL on the labeled cases peaks at 0.463 and collapses to 0.133, and RL on a static pool of generated tasks falls from 0.441 to 0.359. The scaling law proves that with more compute, the self improving loop can scale a model further than a static training data can do. This is a positive finding for the community of clinical and self improving research.

The scaling law is also a new finding in self improving domain. Prior works show Self-play plateaus or degrades after a few iterations [2, 5], and repeated training on generated data degrades models [9]. To our knowledge, the only scaling law for self-play [5] is in Lean theorem proving, where a compiler checks every proof, and is measured on the training problems. No prior methods reports accuracy against training compute.

  1. We show the scaled solver passes frontier models. RSIMed-27B beats every GPT-5.6 configuration; the best scores 0.399 at 38 times its inference cost. The solver's own tasks also train it better than tasks from four stronger teachers, including GPT-5.5: 0.350 against at most 0.344.

We will integrate the proposed changes into the final manuscript.

References

[1] Zhao, et al. Absolute Zero: Reinforced Self-play Reasoning with Zero Data.

[2] Huang, et al. R-Zero: Self-Evolving Reasoning LLM from Zero Data.

[3] Liu, et al. SPICE: Self-Play In Corpus Environments Improves Reasoning.

[4] Li, et al. WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvement.

[5] Bailey, et al. Scaling Self-Play with Self-Guidance.

[6] Liu, et al. Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain.

[7] Chen, et al. HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs.

[8] Huang, et al. Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMs.

[9] Shumailov, et al. The Curse of Recursion: Training on Generated Data Makes Models Forget.

AuthorsOpenReview ↗

Thanks for the review

Dear Reviewer qy7x,

Thanks for spending the time to write the review. As the discussion period ends today, we hope our response addresses your concerns. We are happy to take any follow up clarifying questions you may have.

Reviewer_WqcLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RSIMed, which uses retrieved medical references and feedback from a diagnostic model to continuously generate training questions. The model then learns from these questions through reinforcement learning.

Critical strengths

The results suggest that the generated tasks provide useful training beyond the supervised warm start. This is the strongest evidence supporting the practical value of the approach.

Critical weaknesses

-I think the method is adaptive training on generated data, rather than recursive self-improvement. The solver is updated, but the proposer and judge remain frozen, and the proposer only receives updated performance history. This creates a feedback loop, but it is unclear whether the system becomes better at generating useful supervision as the solver improves. Please clarify the definition of RSI used here and compare against continuous task generation without solver feedback under matched budgets.

  • I am not convinced that generating additional questions addresses the clinical data limitation described in the introduction. More questions do not necessarily mean more useful clinical information. The generated tasks may provide additional practice with existing medical knowledge, but they do not replace diverse patient cases with reliable diagnoses. Please clarify what information these tasks add beyond the original cases and retrieved references, and whether the benefit comes from question generation or simply additional training on the retrieved knowledge.

  • RSIMed receives more generated tasks throughout training, while the baseline repeatedly uses a smaller fixed pool. Therefore, the improvement could come from having more fresh and diverse data, rather than adapting questions to the solver’s weaknesses. A baseline that continuously generates questions without solver feedback, under comparable generation and training budgets, would help answer this. Also, the "MedVLSynther" label in Figure 1 should distinguish this static-pool control from a reproduction of the original method.

  • Appendix B says the validator accepts questions when the retrieved evidence is inconclusive, when retrieval returns nothing, or when the model fails to produce a verdict. Not finding contradictory evidence does not mean that an answer is correct. This seems weaker than the paper’s claim that retrieval addresses unreliable clinical supervision. The independent audit is useful, but it is also model-based and excludes uncertain assessments from the reported validity rate. Please provide clinician-reviewed examples of accepted errors and compare against a stricter validation policy.

  • The strongest MIMIC result comes from the full SFT+RL pipeline, but the HealthBench improvement comes from the RL-only variant. Table 11 shows that the full pipeline actually performs worse than the base model on HealthBench. The paper acknowledges this tradeoff, but it should be more prominent in the abstract and conclusion rather than suggesting that the same model improves on both. Also, "trained only on diagnosis" is not accurate as written, since the generated tasks include treatment, pharmacology, mechanisms, and other clinical topics. The relatively small HealthBench gain also needs uncertainty estimates.

Questions for authors

See above.

  • Minor suggestion: Please keep the most relevant supporting material and cross-reference it clearly. Important details such as the permissive validation policy and the full pipeline’s HealthBench decline should be easier to find in the main paper.
6 comments
AuthorsOpenReview ↗

Thanks for the review and response to questions - Part 1

We thank the reviewer for the detailed and constructive review. We ran the requested controls, and each answer follows the quoted point.

I think the method is adaptive training on generated data, rather than recursive self-improvement. … it is unclear whether the system becomes better at generating useful supervision as the solver improves. Please clarify the definition of RSI used here and compare against continuous task generation without solver feedback under matched budgets.

We thank reviewer for this fundamental question. Following the work cited in our introduction, RSI is a loop in which a model generates the supervision it learns from, with no stronger model and no new labels. In RSIMed every role is a frozen copy of the solver's base model, and the tasks at step tt depend on the current solver through its accuracy and task history. Training it by RL needs a reward for good tasks, and in medicine that reward needs the very correctness check that is missing. We will state this definition in the introduction.

The unique thing we would like to show here is that the self improving configuration, or using the same small model for all roles including proposer, judge, solver, counterintuitively performs better than using a stronger frontier model. This is a finding that challenges prior findings in some works. We see this as a result of smaller model creating more difficult questions on average, which provides a stronger training signal per step.

Under matched budgets, we see feedback makes the solver learn faster. We ran the requested experiments. We see that with the number of fresh tasks matched, feedback still makes the solver learn faster. The results below are done with the RSIMed-9B recipe with no feedback and matched sample sizes. We use the same Qwen-3.5-9B proposer/judge at training time, and GPT-5.5-Chat judge for validation scoring.

RunAccuracyPercentage of task with Learning Signal
RSIMed-9B0.42619.8%
No feedback0.41311.0%

Note the accuracy here is the matched accuracy at no feedback's best step, which is far from the saturation performance. We see that even at matched sample size, RSIMed-9B is still the best model with highest learning signal. We further see that the feedback allows the proposer to propose a higher proportion of useful tasks, at 19.8% vs 11.0% for no feedback variant.

I am not convinced that generating additional questions addresses the clinical data limitation described in the introduction. … Please clarify what information these tasks add beyond the original cases and retrieved references, and whether the benefit comes from question generation or simply additional training on the retrieved knowledge.

We see that directly training on the knowledge, without the model reflecting, reasoning on the data, reaches a much lower accuracy on the downstream tasks. We trained the same supervised Qwen3.5-9B checkpoint for one epoch on all 304,915 passages the generator retrieves for the training cases, 35.2M tokens in total. The table gives each model's change in MIMIC-IV accuracy. Each change is measured with a GPT-5.5-Chat grader.

TrainingAccuracy change
Knowledge only−9.4
RSIMed-9B+18.1

Giving the knowledge at inference does not help either. In Table 11, three RAG methods score 0.301 to 0.329 with Qwen3.6-27B, against 0.332 without retrieval.

A limitation with knowledge only learning is that the knowledge is limited to the database itself, which may not cover all parts of the validation tasks. The self improving loop, on the other hand, can generate questions beyond the database, in a format closer to the downstream tasks, and allows the model to reason and reflect on the results. We split the case-by-case comparison of Appendix J.3 by how often each diagnosis appears among the labeled cases. On the 52 evaluation cases whose diagnosis has no labeled training case, the RL stage raises the accuracy of Qwen3.6-27B from 0.115 to 0.173.

RSIMed receives more generated tasks throughout training, while the baseline repeatedly uses a smaller fixed pool. Therefore, the improvement could come from having more fresh and diverse data, rather than adapting questions to the solver's weaknesses. … Also, the "MedVLSynther" label in Figure 1 should distinguish this static-pool control from a reproduction of the original method.

We would like to further note that the static pool variant fell from its 0.441 peak to 0.359 by step 155, after only 9,920 draws from its 15,000 tasks. Therefore, repeated training is not a cause for its collapse, and even if it has 43k tasks, it will still collapse well before all tasks are consumed.

We will rename the Figure 1(b) curve "Static pool (MedVLSynther setting)" and describe it as a control built with our proposer.

AuthorsOpenReview ↗

Thanks for the review and response to questions - Part 2

Appendix B says the validator accepts questions when the retrieved evidence is inconclusive, when retrieval returns nothing, or when the model fails to produce a verdict. … The independent audit is useful, but it is also model-based and excludes uncertain assessments from the reported validity rate. Please provide clinician-reviewed examples of accepted errors and compare against a stricter validation policy.

We see that wrong keys stay rare and carry little learning signal. We audited a random sample of 100 accepted short-answer tasks from each fifth of training in RSIMed-27B and RSIMed-9B with GPT-5.5-Chat as automatic judge. We see that:

  • They do not accumulate. The valid share rises from the first fifth of training to the last in both runs: from 78.2% to 86.5% in RSIMed-27B and from 72.0% to 80.0% in RSIMed-9B.

  • Invalid tasks usually do not get any learning signal. 73% of invalid tasks get no rollout that matches the key, against 54% of valid tasks, so they give no correctness signal.

  • We further replayed a strict version of the validator, which accepts a task only when the retrieved evidence supports its key, with a frozen Qwen3.5-9B checker. The strict version raises the valid share from 81% to 88% but rejects 19% of the valid tasks. As a comparison, the paper's validator keeps 98% of the valid tasks and rejects 11% of the invalid tasks. In other words, requiring support removes 2.3 units of valid signal for every unit of invalid signal.

Due to the amount of tasks and time constraint, we will include clinician's review on invalid or uncertain tasks in the final manuscript. We will add them to the appendix with the clinician's verdict.

The strongest MIMIC result comes from the full SFT+RL pipeline, but the HealthBench improvement comes from the RL-only variant. … it should be more prominent in the abstract and conclusion rather than suggesting that the same model improves on both. Also, "trained only on diagnosis" is not accurate as written … The relatively small HealthBench gain also needs uncertainty estimates.

Thanks for the advice. We will move the training-stage table into the main text. The abstract and conclusion will say that the full pipeline is best on MIMIC-IV and RL only is best on HealthBench. "Trained only on diagnosis" becomes "trained only on tasks seeded by rare-disease admissions".

We would like to point out that the main focus of the work, and all improvements on the self improving loop, is on the RL part of the pipeline. Therefore, RL only result being better actually indicates that the self improving pipeline gives a more generalizable OOD performance that normal SFT on generated data does not provide, which strengthens our claim.

On the uncertainty note, we see that RL only is at or above the untrained model in every category with zero targeted training data for HealthBench Pro:

Good faithRed teamingTypicalDifficultConsultationResearchWriting
+0.010+0.017+0.001+0.023+0.016+0.001+0.017

With that said, we will further adjust the claim to make this more clear.

Minor suggestion: … Important details such as the permissive validation policy and the full pipeline's HealthBench decline should be easier to find in the main paper.

We will give the validator's full acceptance rules in Section 3.3 and move the training-stage table into the main text.

For reference, the validator's full prompt is below. The Options line appears only for multiple-choice tasks.

System prompt:

You are a medical fact-checker. You are given:
(1) A candidate training question + its proposed answer (+ options if MCQ).
(2) Retrieved passages from a medical knowledge database (re-queried using the candidate question).

Decide whether the question's proposed answer CONTRADICTS the retrieved knowledge.

BE LENIENT — only reject obvious contradictions:
- If the retrieved knowledge directly and unambiguously STATES something that makes the proposed answer WRONG → "contradict"
- If the retrieved knowledge is silent, tangential, or only partially relevant → "ok"
- If the question is about something NOT in the retrieved passages (out-of-knowledge) → "ok" (we accept new knowledge)
- If the question is well-formed but the answer seems questionable without direct contradiction from the passages → "ok"
- If the question is poorly formed / ungrammatical / incoherent → "contradict"

Keep any internal reasoning UNDER 500 WORDS, then output ONLY a JSON object with a one-sentence reason. No markdown, no explanation.
{{"verdict": "ok" or "contradict", "reason": "..."}}

User message:

Question: <generated question>
Options: <options as JSON>
Proposed answer: <proposed answer>

Retrieved passages from the database:
[<source>] <passage 1>

[<source>] <passage 2>

[<source>] <passage 3>

Does the proposed answer CONTRADICT the retrieved knowledge?
AuthorsOpenReview ↗

Thanks for the review

Dear Reviewer WqcL,

Thanks for spending the time to write the review. As the discussion period ends today, we hope our response addresses your concerns. We are glad to take any follow up clarifying questions you may have.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.