- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper tries to probe the causal factors of the evidence preference of LLMs in medical situations. It intends to probe why LLMs prefer a certain set of evidence. For this, the authors propose the PMC-ConForm dataset which are kind of based on "virtual patients" synthesied from two patients.
Critical strengths
The general idea to improve the stability in medical AI and to reduce bias is valid. And it's also very important.
Critical weaknesses
Low Probability Coincidence in the Reported Data
It seems that the numerical values in this paper can be recalculated from the percentages. This is of extremely low probability due to rounding. The rounding function is not an injective mapping, it has no inverse mapping and breaks the multiplicative group.
Take the first line of Table 9 as an example. . (2.625 rounds to 2.63 in Microsoft Excel, to 2.62 in R or Python. Manually it's usually 2.63 since rounds half up is the standard in K-12.). So it recalculates to 2.63. This is of low probability because and . So with , for there are 6 possible values, 2.60, 2.61, 2.62, 2.63, 2.64, 2.65, satisfying . The coincidence that the measured value can be accurately recalculated from the percentage is only .
This coincidence is found for all 8 rows in Table 9. All the measured are accurately recalculated from the rounded percentage. .
Similar patterns are found in Table 1 and Table 10, where the FPS-nats-Frozen-unseen values of all models can be calculated from the seen values by multiplying by an integer percentage coincidentally.
| Model | Table 1 | Table 10 | Multiplier |
|---|---|---|---|
| MedGemma-4B | 1.7131 | 1.9187 | 112% |
| Gemma3-4B | 2.9976 | 3.5372 | 118% |
| HuatuoGPT-o1-7B | 0.8178 | 0.8914 | 109% |
| Qwen2.5-7B | 2.2400 | 2.5760 | 115% |
| BioMistral-7B | 0.2656 | 0.3240 | 122% |
| Mistral-7B | 0.3867 | 0.4486 | 116% |
| Llama3-8B | 0.7883 | 0.9775 | 124% |
| Qwen2.5-14B | 3.6525 | 4.1273 | 113% |
Take MedGemma-4B as an example: . This is also of very low probability because . So 1.9187 is the only four-decimal number in that would lead to this coincidence.
Shrink Lines Are Not Calculated from the raw data
In section F.2, the authors define that:
let $r = \frac{S_{ours}}{S_0}. The matched shrink line is .
Hence we have . Take MedGemma-4B as an example: , which is pretty far from the reported 0.7195. However, if we use the , we have . So it seems that the shrink lines are calculated from the percentage instead of the raw data. This is not standard for science, you should calculate all numerical values with the Double type and only round them before writing the paper to prevent the amplification of the rouding errors.
Research Problem is Invalid - Part I
The problems of medical diagnosis have ground truth values. These ground truth values are not hard to get. In this case, the reason for adopting the invariant goal instead of the ground-truth accuracies is strange. IMO, it's not supported in this submission.
Take an autonomous driving algorithm as an example. For a situation that the car encountered a red light, it's clearly required by the law that the car should stop. So, in this case, there is a ground truth for the autonomous driving algorithm, and it should brake. For this situation, we simply aim to improve the decision accuracy. There's no need to improve the invariance or stability. Actually, it's the logical implication of accuracy. If the car always makes the correct decision, it will be stable. Whereas, in the situation where our car is behind a slow-moving car and an overtaking decision is needed, there's no universal ground truth on whether the algorithm should overtake or not. In this case, it's extremely hard to get a ground truth label, and we have to pursue the stability of the algorithm such that, in similar situations, it is expected to have similar behaviors.
I'm not claiming that stability against input formats is not important. However, we can simply investigate this stability against real instances with real ground truth labels. We can study whether the accuracies drop when we alter the format. Also, if the accuracy is high enough, the stability is implied. I don't see any reason to abandon the ground truth instances and pursue the virtual ones.
Moreover, when a specific format causes incorrect diagnoses, our priority should be fixing correctness, not consistency, since consistency can be artificially inflated by making correct diagnoses wrong. Practically, if a format improves accuracy, we can simply use prompt engineering to guide physicians to adopt it. Consider a calculator on Windows, Mac, or Linux: it ignores inputs like 'o,n,e,p,l,u,s,o,n,e.' While returning 't,w,o' would be nice, the lack of support for this format does not stop it from being an excellent calculator.
Research Problem is Invalid - Part II
Medical diagnosis is essentially computing quotients of sets. The essence is to identify subsets from the universal set based on certain evidence. That's by definition. For an arbitrary universal set, there are multiple partitions and each partition can be correlated with certain evidences. For instance, for , one "diagnosis" is to diagnosis whether a number is odd or even. This partition is the quotient set . The evidence used for this diagnosis is:
However, if the diagnosis is about whether a number is divisible by 3, the partition becomes and the evidence you need becomes
There's no unified "evidence" for medical diagnosis. Different evidence is needed for different subsets. Overlapping subsets exist. For instance, and they rely on different evidence . The whole diagnosis system is not a set partition, as it doesn't adhere to the definition , It's the diagnosis of each dieases the partitions. And they each has their own quotient mapping which by definition indicate the evidence.
A condition qualifies as a disease precisely because specific evidence (the quotient map) allows us to stably partition patients into a quotient set, where those within the same equivalence class share similar treatment approaches. Thus, the quotient set (the disease) is induced from the quotient map (the evidence).
If the evidence were left undefined, merely showing a statistical correlation with the disease, then (as implied by the limitations section of this paper) an unmeasured confounding variable would have been overlooked, constituting a limitation of the study. However, I argue that this is not the case here. Rather, the definition of the disease itself provides the very evidence it requires. The quotient sets correspondes to the quotient maps: . This circularity in the evidence casts doubt on the validity of the entire research question. It's somewhat by definition. The whole research question is somewhat tautological. For instance, an AIDS patient is an AIDS patient because it's defined as , , . Hence the definition of AIDS requires microbiology as this is inherently defined in .
Diseases for which are not defined on pathology and microbiology evidence but instead on physical exam, physiology, and self-report include Myopia, Color Vision Deficiency, Pes Planus, Hypertension, Malocclusion, Erectile Dysfunction, Premature Ejaculation, etc.
Model Obsolescence
The models used in this paper include Qwen 2.5, MedGemma, and Mistral 7B. They are mostly old models released between 2023 and 2025.
These models all predate or are contemporary to DeepSeek-R1. From a technical standpoint, they all seem to be pre-R1 models. Nowadays, most people are using post-R1 models that are able to think with those super long reasoning chains. I doubt whether the format will have this much influence on those post-R1 models because these modern models will repeat the information in natural language and reason about each of your provided records carefully. For difficult diagnoses, they can reason for tens of thousands of tokens. Will they still be this sensitive to format after this long reasoning chain? I think the authors have to conduct experiments with post-R1 models to demonstrate the reported findings.
I also find the following sentence in the prompt (page 17):
Read the records and answer with a single letter. Do not explain your reasoning.
So it seems that the authors are actually asking the models not to think. Bad experimental design, in my opinion. Nowadays, almost all models utlizes GRPO and can produce extra-long reasoning behaviors. These old models that cannot reliably generate long chains of thought have largely been phased out.
Furthermore, long reasoning has significantly improved the performance of LLMs across most tasks. By 2026, almost all new models are those with long reasoning. In the context of medical diagnosis, enhancing diagnostic performance is clearly the top priority. I note that the training method proposed in this paper is inherently suited for models without reasoning processes. If one were to actively abandon long chain-of-thought techniques, which could substantially boost performance, in pursuit of this so-called stability, I would consider that a case of putting the cart before the horse.
Questions for authors
Why are the seeds used for experiments 20260908 and 20260915? Are they dates?
Why do the experiments in the Motivation section use the larger seed 20260915 while the experiments in the Experiments section use the 20260908 one?
Why not give more examples?
9 comments
Weakness 3
A: Thank you for this thoughtful comment! We agree that different diseases are defined by different evidence. For this reason, our paper does not judge which evidence a diagnosis should rely on, but studies whether a model's preference changes when only the presentation of the same evidence changes.
(1) This question is well-posed whichever evidence defines a disease. Each comparison holds the evidence, candidate diagnoses, and positions fixed and changes only the layout of the records, so the patient and the definition of each disease stay the same. In your notation, takes the same value whether the HIV test result is written as a sentence or as a JSON field, and so does whether blood pressure measurements appear in prose or in a table. The appropriate answer should therefore not depend on the layout. Yet for 9.8%–17.1% of conflicts, some layout change reverses the frozen model's preference (Sec. 3).
(2) The source ranking motivates this question but is not its premise. Observation 1 shows that models weigh evidence sources systematically, which raises the question of whether this weighting depends on presentation. The ranking describes how models weigh sources on PMC-ConForm rather than a clinical evidence hierarchy, and we have made this scope explicit in Observation 1 and the abstract. As you point out, case reports may over-represent diagnoses confirmed by pathology or microbiology, so the ranking may differ for conditions diagnosed mainly by physical examination or physiological measurement. We now note this in the Limitations. Neither our metrics nor FORMEDER uses this ranking: FORMEDER's target is each model's own cross-format consensus for each instance.
Weakness 4
A: Thank you for this insightful comment! It raises an important question for practical use: whether format sensitivity persists under long reasoning.
We originally chose these models for two reasons. First, our preference score is computed from answer probabilities, and FORMEDER fine-tunes the model with LoRA, so both require open-weight models. Second, to test whether medical adaptation or model size improves format robustness, we paired three medical models with their general-purpose base models (MedGemma-4B/Gemma3-4B, HuatuoGPT-o1-7B/Qwen2.5-7B, BioMistral-7B/Mistral-7B) and added Llama3-8B and Qwen2.5-14B. When we selected the models, MedGemma-4B/Gemma3-4B was the most recent such pair. Neither medical adaptation nor model size substantially raises frozen FCC (Tab. 1 of the paper).
We would like to clarify that the single-letter instruction was introduced for reliable scoring rather than to suppress reasoning. In preliminary runs without this instruction, models often explained first and did not always end with a clear option letter, so their answers could not be parsed consistently across models and formats. In this direct-answer setting, a single letter at the first answer position also provides the option-letter probabilities used by the preference score, and greedy decoding gives a deterministic score for each input (App. B.4).
To address your concern, we further enabled reasoning for HuatuoGPT-o1-7B, a medical model trained for complex reasoning, and Qwen3-8B, a post-R1 model, using the same prompts on the full test set. With reasoning enabled, the single-letter instruction only constrains the final answer, and both models still reason before giving it. Each model uses its own generation configuration, so Qwen3-8B uses sampling in this mode, as its developers recommend against greedy decoding with reasoning. The results are as follows.
Table 3. Main metrics without and with reasoning.
| Model | Reasoning | FPS↓ | FCC↑ (%) | RA↑ (%) |
|---|---|---|---|---|
| HuatuoGPT-o1-7B | off | 0.8178 | 81.83 | 40.62 |
| HuatuoGPT-o1-7B + FORMEDER | off | 0.0336 | 93.86 | 43.32 |
| HuatuoGPT-o1-7B | on | 2.1210 | 81.66 | 56.09 |
| HuatuoGPT-o1-7B + FORMEDER | on | 0.2463 | 88.87 | 59.02 |
| Qwen3-8B | off | 2.1818 | 88.68 | 28.99 |
| Qwen3-8B + FORMEDER | off | 0.0841 | 95.74 | 31.62 |
| Qwen3-8B | on | 2.6774 | 79.50 | 35.31 |
| Qwen3-8B + FORMEDER | on | 0.3188 | 88.57 | 37.05 |
As shown, reasoning improves accuracy but does not remove format sensitivity. With reasoning, frozen FCC is 81.66% for HuatuoGPT-o1-7B and 79.50% for Qwen3-8B, no higher than without reasoning. A likely reason is that, even when a model restates the records in its reasoning, which details it restates and how it weighs them can still depend on how the records are presented. FORMEDER remains effective in this setting. Although trained without reasoning, it raises FCC to 88.87% and 88.57% and improves RA for both models. This suggests that FORMEDER reduces how much the model's judgment depends on the format, so the effect carries over when the model reasons before answering. We have added these results to the revised manuscript.
Question 1 & 2
A: Yes, both seeds are dates in YYYYMMDD form. Before submission, we had an LLM-based agent rerun our experiments with fixed seeds to ensure reproducibility, and the agent chose these dates as the seeds. 20260908 is the training seed for all trained models (Tab. 8), and 20260915 is the resampling seed for the bootstrap intervals in the analyses (App. C.1). The Motivation section analyzes frozen models without training, so only the resampling seed appears there.
Question 3
A: Thank you for this helpful suggestion! In the revised manuscript, we have replaced the original case study with five complete examples from the PMC-ConForm test set: two conflicts with a format-induced reversal, two without, and one failure case of FORMEDER. Each shows the full records in prose and in a structured format, with the answers and preference scores of MedGemma-4B before and after FORMEDER under all four position configurations.
We hope our responses clarify the concerns raised. We have revised the manuscript accordingly. Thank you again for your careful review!
Dear Reviewer xyf4, we thank you for the detailed comments. Below, we address each of your concerns.
Weakness 1 & 2
A: Thank you for your careful review! We immediately examined the relevant records on our local computers and servers and identified a problem with the handling of intermediate precision in our agent-assisted result-processing workflow: these entries were derived from rounded values rather than computed from the full-precision results. The AI Use Statement of the revised manuscript describes the role of these agents in more detail.
(1) Cause. The agent that processed the evaluation outputs saved the full-precision results in a temporary directory outside our project directory. The table-generation agent could not locate them and used the rounded values of a draft summary as exact inputs: 100r rounded to an integer (the 100r column of Tab. 9) and the frozen model's unseen/seen FPS ratio rounded to two decimals. It then derived the entries in Table 1 through Sours=rS0 and FPS0,unseen=FPS0,seen× this ratio, carrying the rounding errors into them.
Table 1. Affected entries (r~=round(100r)/100).
(2) Original values. We manually recovered the original values from this temporary directory, as listed below.
Table 2. Unrounded S and FPS values (single precision).
Applying the steps in Table 1 to these values reproduces the submitted entries.
(3) Impact. Among the 32 affected entries, the largest relative difference is 0.97%. In your MedGemma-4B example, Sours=2.6208 rather than 2.63, and the shrink line is (2.6207688/6.2464542)×1.7131476=0.7188 rather than 0.7195. No conclusion changes: FORMEDER retains 33%–58% of frozen prose strength, its FPS remains 5.8–14.2× below the matched shrink line, and every backbone still has lower unseen-format FPS than its corrected frozen reference in Tab. 10.
(4) Revision. We have recomputed Tab. 9 and the corresponding range in App. F.2, and replaced the frozen unseen-format FPS in Tab. 10. All derived quantities are now computed at full precision and rounded only for display, and Tab. 9 reports S0 and Sours to four decimals so that r and the shrink line can be recomputed from the table. We also fixed the RA aggregation. In some evaluation runs, one single-record reply was missing, and the aggregation excluded it instead of scoring it as an error, so these runs used a denominator of 5,287 rather than 5,288. All RA values now use the same denominator of 5,288 (App. F.1), which lowers the affected entries in Tab. 1 and Tab. 2 of the paper by 0.01, with rankings and conclusions unchanged. All tables in this response use corrected values.
We fully recognize the importance of numerical consistency and reproducibility in empirical research, and we sincerely apologize for these errors in the submitted tables. Thank you again for helping us identify and correct this issue.