- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper treats six-level misinformation detection on the PolitiFact scale as ordinal regression rather than flat classification. The proposed BSOR is a plug-in output head built on CORN, with three additions: a loss weighted by misprediction distance, label-based sample weights that balance the subtasks, and a KL soft-label regularizer. On RDE and LIAR, replacing the flat head with BSOR reduces MAE for most of nine encoders and six detection methods.
Critical strengths
- The motivation is clear: fact-checking labels are ordered, and flat classification ignores this order.
- The head is simple to plug in, and it is tested across a wide range of encoders and detection methods.
- The error-distribution analysis shows that the gains mainly come from reducing predictions that are far from the true label.
Critical weaknesses
W1. The relevance of the method to current misinformation detection is unclear.
BSOR changes only the output head and loss of a classifier, and none of its components uses properties specific to misinformation (§4); the same head could be applied to any task with ordered labels. All experiments use encoder-based models without LLMs, such as BERT, RoBERTa, DeBERTa, dEFEND, and GEAR (Tables 2 and 3), and no LLM-based detector or LLM baseline is included. Current misinformation detection increasingly relies on LLMs, which may already capture the ordering of truthfulness labels, so it is unclear whether BSOR remains useful in this setting. A comparison with LLM-based methods, or an application of BSOR to an LLM-based detector, would help establish its significance.
W2. BSOR is not compared with existing ordinal regression methods.
The main results (Tables 2 and 3) compare BSOR only against flat classification heads. CORAL, which the paper cites (§2.2), is not compared, and the KL soft-label regularizer (§4.5) closely resembles soft ordinal labels [1], which is neither cited nor compared. The ablation (Table 4) suggests that most of the gain comes from CORN itself, as the authors also note (§5.4). For BERT-base, for example, CORN reduces MAE from 0.7402 to 0.7031, while the full BSOR reaches 0.6978. Since this ablation covers only three encoders on RDE, the added value of BSOR beyond standard ordinal regression remains unclear.
W3. The claim of consistent improvement may need qualification.
The abstract states that BSOR “consistently reduces MAE for nine general-purpose encoders.” This holds on RDE, but on LIAR, BSOR increases MAE for BiLSTM+Att (+6.56%) and WordAvgMLP (+2.95%) (Table 2). The authors explain that the flat baselines in these two cases collapse to a constant prediction (§5.2), which is reasonable, but the abstract’s wording does not reflect this exception. In addition, some gains are within one standard deviation (e.g., DeBERTa+LSTM: 0.5021±0.0044 vs. 0.4991±0.0057), and no significance tests are reported.
W4. The evaluation lacks class-level metrics, which leaves two claims unsupported.
The evaluation uses only MAE and Off-1 Accuracy (§5.1). Without accuracy, macro-F1, or per-class results, two points remain unclear. First, it is unclear whether BSOR trades exact accuracy for smaller errors: Figure 2(a) shows that exact-match predictions decrease for five of nine encoders on RDE and increase for the other four. Second, the paper argues that higher-label classes dominate training (§4.4), but provides no per-class evidence that this bias harms performance or that the corresponding component corrects it. That component also brings only small gains in the ablation (e.g., 1.0074 vs. 1.0073 for TextCNN+GloVe, Table 4).
The paper contains several typos, e.g., “Supplementar” (Figure 1, line 166), and missing spaces in “detection(Zhou” (line 094), “r_qi .The” (line 216), “assumption.To” (line 227), “bias.To” (line 261), and “methods.Six” (line 378).
[1] Raúl Díaz and Amit Marathe. Soft Labels for Ordinal Regression. In Proceedings of CVPR, 2019.
Questions for authors
See weakness.
6 comments
Response to W1 (2/2): Comparison with LLM-based methods
W1 (2/2): Comparison with LLM-based methods
③ Comparison with LLM-based methods. We agree that this comparison is necessary and have added it from two angles.
First, LLMs in direct question answering versus the best detection methods. We tested the direct question-answering capability of LLMs on both datasets: the locally deployed Qwen2.5-7B-Instruct with greedy decoding, and the API flagship model GLM-5; the output is restricted to a single digit from 0 to 5 corresponding to the six class labels. The controls are the best detection method on each dataset: dEFEND+BSOR on RDE and Speaker context+BSOR on LIAR. The results are as follows; the error distribution is the percentage of test samples in each off-by-k bin:
| Dataset | Model / Method | MAE | off-by-0 | off-by-1 | off-by-2 | off-by-3 | off-by-4 | off-by-5 |
|---|---|---|---|---|---|---|---|---|
| RDE | dEFEND+BSOR | 0.3298 | 84.26% | 5.03% | 7.04% | 0.82% | 2.78% | 0.05% |
| RDE | Qwen2.5-7B 0-shot | 0.6341 | 58.08% | 25.70% | 12.01% | 3.26% | 0.80% | 0.14% |
| RDE | GLM-5 0-shot | 0.3615 | 68.24% | 27.72% | 3.73% | 0.27% | 0.02% | 0.02% |
| LIAR | Speaker context+BSOR | 0.9083 | 46.84% | 29.31% | 13.48% | 7.17% | 2.88% | 0.23% |
| LIAR | Qwen2.5-7B 0-shot | 1.5378 | 18.16% | 38.97% | 22.76% | 13.25% | 4.75% | 2.10% |
| LIAR | GLM-5 0-shot | 1.1777 | 28.76% | 39.67% | 20.42% | 7.64% | 3.20% | 0.31% |
Direct question answering does not reach the level of the best fine-tuned detection methods on either dataset: on RDE, GLM-5 achieves an MAE of 0.3615 and Qwen2.5-7B achieves 0.6341, both higher than the 0.3298 of dEFEND+BSOR; on LIAR, GLM-5 achieves 1.1777 and Qwen2.5-7B achieves 1.5378, both higher than the 0.9083 of Speaker context+BSOR. The error distributions of the LLMs also shift to the right as a whole, and long-range errors remain severe. This shows that for fine-grained misinformation detection, which requires domain supervision and ordinal structure modeling, fine-tuned detection methods still have a clear advantage. This part is not reported in the main text because it is difficult to fit into the existing tables and due to space limitations.
Second, BSOR is equally plug-and-play for LLM backbones. We fully fine-tune Qwen2.5-1.5B and replace the Flat head with BSOR: on RDE the MAE decreases from 0.6236 to 0.6037, a relative reduction of 3.19%, and on LIAR from 1.3455 to 1.2445, a relative reduction of 7.51%; Off-1 Accuracy improves by 1.28 and 1.92 percentage points respectively, and both improvements are statistically significant. Its misjudgment-distance behavior is consistent with the pre-trained encoders. The Qwen2.5-1.5B results have been added to the encoder experiments in the revision; the direct question-answering comparison is new in this response. The details are given in Table 2 and Section 5.2 (page 7) of the revision.
In summary, the best existing detection methods outperform LLMs in direct question answering, and the BSOR head itself can be directly transplanted to LLMs with consistent improvements.
Response to W2 (1/2): Comparison with ordinal regression methods and the KL soft-label regularizer
W2 (1/2)
① Comparison with other ordinal regression methods. To demonstrate the advantage of our method over existing methods for multi-class misinformation detection, and the necessity of our adaptation-oriented improvements, we added comparison experiments with representative ordinal regression baselines, covering three families: CORAL (latent-variable thresholding), CDW-CE (cost-sensitive weighting), and EMD$^2$ (distribution matching). The experiments cover 6 settings: three encoders (RoBERTa-base, BERT-base, and DistilBERT) on the RDE and LIAR datasets. All methods are trained and evaluated with exactly the same training protocol as the Flat head. MAE is the mean of three random seeds; bold marks the best in each row.
| Dataset | Encoder | Flat | CORAL | CDW-CE | EMD$^2$ | BSOR |
|---|---|---|---|---|---|---|
| RDE | RoBERTa-base | 0.5910 | 0.5783 | 0.6014 | 0.5598 | 0.5549 |
| RDE | BERT-base | 0.7402 | 0.7299 | 0.7917 | 0.6982 | 0.6978 |
| RDE | DistilBERT | 0.7583 | 0.7609 | 0.7915 | 0.7421 | 0.7286 |
| LIAR | RoBERTa-base | 1.3856 | 1.1795 | 1.2315 | 1.2970 | 1.2897 |
| LIAR | BERT-base | 1.3154 | 1.1943 | 1.2016 | 1.2777 | 1.2603 |
| LIAR | DistilBERT | 1.3276 | 1.1943 | 1.2484 | 1.3011 | 1.2933 |
In the table, BSOR achieves lower MAE than Flat in all 6 settings, with an average reduction of 4.91% and an average Off-1 Accuracy improvement of 2.32%, verifying the effectiveness of BSOR over the flat classification baseline. Some ordinal baselines achieve lower MAE than BSOR on LIAR, which looks better; however, the new class-wise performance analysis shows that the gains of CORAL and CDW-CE come from biased prediction distributions: they concentrate predictions on the middle levels at the cost of abandoning the recognition of the boundary levels. The table below shows the per-class Recall results from that analysis: for RDE we excerpt the most biased setting (BERT-base), and for LIAR we give all three encoder settings. The Method column lists the four output heads Flat, CORAL, CDW-CE, and BSOR; values are percentages. The complete tables including EMD$^2$ and F1 are given in Appendix F:
| Dataset | Encoder | Method | ||||||
|---|---|---|---|---|---|---|---|---|
| RDE | BERT-base | Flat | 58.5 | 34.7 | 38.8 | 30.1 | 74.0 | 55.5 |
| RDE | BERT-base | CORAL | 56.1 | 33.3 | 36.7 | 25.9 | 69.5 | 56.6 |
| RDE | BERT-base | CDW-CE | 24.5 | 38.9 | 32.7 | 36.8 | 63.8 | 30.3 |
| RDE | BERT-base | BSOR | 56.4 | 36.4 | 42.3 | 37.0 | 70.9 | 58.2 |
| LIAR | RoBERTa-base | Flat | 28.3 | 23.3 | 30.7 | 23.7 | 35.1 | 18.5 |
| LIAR | RoBERTa-base | CORAL | 1.8 | 6.3 | 42.5 | 47.7 | 30.3 | 1.3 |
| LIAR | RoBERTa-base | CDW-CE | 0.0 | 8.0 | 36.1 | 47.3 | 26.8 | 0.0 |
| LIAR | RoBERTa-base | BSOR | 21.7 | 26.1 | 29.3 | 29.6 | 33.7 | 20.2 |
| LIAR | BERT-base | Flat | 12.0 | 31.9 | 22.4 | 27.7 | 45.4 | 18.0 |
| LIAR | BERT-base | CORAL | 0.4 | 3.2 | 29.1 | 62.3 | 23.4 | 0.9 |
| LIAR | BERT-base | CDW-CE | 0.0 | 7.1 | 30.1 | 54.4 | 31.7 | 0.0 |
| LIAR | BERT-base | BSOR | 12.7 | 26.9 | 31.6 | 29.1 | 37.8 | 19.4 |
| LIAR | DistilBERT | Flat | 13.4 | 31.3 | 29.1 | 19.6 | 47.1 | 19.6 |
| LIAR | DistilBERT | CORAL | 1.8 | 7.1 | 41.1 | 48.3 | 29.5 | 1.9 |
| LIAR | DistilBERT | CDW-CE | 0.0 | 14.7 | 37.1 | 37.5 | 31.2 | 0.0 |
| LIAR | DistilBERT | BSOR | 12.7 | 27.1 | 26.0 | 34.1 | 32.5 | 16.0 |
The biased pattern is directly visible in the table. In all three LIAR settings, CORAL compresses the recall of the two boundary levels and to below 2 percentage points while piling predictions onto the middle levels; the recall of and for CDW-CE drops to zero in all three LIAR settings, and in the BERT-base setting on RDE it also compresses the recall of and by 34.0 and 25.2 percentage points relative to Flat. The apparent gains of both methods come at the cost of abandoning the recognition of boundary levels, which is not a desirable direction.
The details are given in Section 5.3 and Table 4 (pages 8–9) of the revision; the class-wise evidence is given in Appendix F (from page 33).
② The KL soft-label regularizer. We acknowledge that the KL soft-label regularization scheme used in this section closely resembles existing soft ordinal label methods, and the missing citation was a writing oversight. This module is directly reused from an existing scheme without innovative modification; its purpose is only to stabilize training and reduce mid-training performance jumps and convergence difficulty, and it brings no substantial performance gain. It is therefore only briefly mentioned in Section 4.5 and discussed together with the total loss, without being set as a separate ablation item. We have added the citation in the revision and a short note in the first paragraph of Section 4.5, clarifying that this regularizer comes from existing work and is not a contribution of this paper.
Response to W1 (1/2): Properties specific to misinformation and applicability to ordered-label tasks
Thank you very much for taking the time to review our paper and for the suggestions. Our responses and the corresponding revisions are given below. Due to space limitations, we summarize the theoretical analysis and the experimental improvements, so that the changes and results can be understood without reading the revised manuscript. If you would like further details, we have marked the corresponding locations in the revision.
W1 (1/2): No properties specific to misinformation, and applicability to any ordered-label task
① No properties specific to misinformation. The scientific goal of this paper is to make an initial verification of whether the ordinal regression paradigm itself is effective for fine-grained multi-class misinformation detection. For this purpose, we aimed to stay as close as possible to the native mathematical principles of ordinal regression, and we deliberately did not introduce overly complex customized network modules. Designing a completely novel and complex module from scratch would make the source of the gains hard to isolate: one could not tell whether the performance improvement comes from the ordinal regression paradigm itself or from the added complex architectural components.
Meanwhile, although we did not capture any misinformation-specific properties, we did adapt CORN to the practical needs of multi-class misinformation detection. This part was not clearly stated in the original manuscript, so the third paragraph of the Introduction in the revision now clarifies the relevance of BSOR to misinformation detection. Specifically, multi-class detection rates claims on an ordered scale, and the harm of a misjudgment grows with the level distance. This implies two ideal constraints on the training objective: the cost constraint, that the penalty on a misjudgment should increase strictly with the rank distance; and the stability constraint, that ordinal labels encode only the order of severity levels and not the importance of samples, so the loss aggregation under conditional decomposition should not structurally favor samples with higher label indices. In Appendix C, we further prove theoretically that neither flat classification nor vanilla CORN can satisfy these two constraints.
② The same head could be applied to any task with ordered labels. The two constraints fit ordered rating tasks whose cost is symmetric in direction and monotonically increasing with distance, and whose labels encode order rather than importance. Once the labels imply a direction-asymmetric cost, threshold-type decision boundaries, or the priority of the samples themselves, at least one of the two constraints no longer holds. We also note that fields such as medicine and credit rating do involve these two constraints, so our method is indeed not limited to misinformation detection; its actual effect on other tasks may depend on further properties and remains to be examined. This does not change our original motivation: these modifications were made to address misinformation detection.
The comparison with LLM-based methods is given in the next comment.