- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces InsClaimBench, a benchmark for insurance claim adjudication that evaluates a decision chain from atomic rules to modules, payout decisions, and final payout amounts. The benchmark uses expert-defined logic to construct reference answers and LLM-generated claim files, including controlled factual variants. It contains 3,780 cases, 375 case families, and 3,405 base–variant pairs.
Experiments with six LLMs show that relatively high atomic-rule accuracy does not necessarily translate into reliable case-level judgments, payout calculations, or consistency under factual changes.
Critical strengths
- Deterministic reference payout amounts reveal a substantial gap between payout-decision accuracy (74–80%) and joint decision–amount accuracy (48–73%).
- The large collection of base–variant pairs enables evaluation of consistency under factual changes through metrics such as PAF and MME.
- The benchmark covers multiple insurance domains, uses expert-defined decision logic, and includes independent expert validation.
Critical weaknesses
W1.
Section 4.2 describes payout calculation using the claim file and predicted payout decision, while Appendix A states that human experts also receive applicable calculation rules. It is unclear whether models have access to the same information, making the payout evaluation conditions difficult to interpret.
Payout errors may also arise from eligibility judgments, rule application, or arithmetic calculations. Separating these sources would clarify what capabilities are being evaluated.
Two metric-related issues also deserve attention:
- Joint accuracy combines payout decisions and amount accuracy by assigning zero to predicted denials. Amount accuracy should also be reported for truly payable cases correctly predicted as payable.
- MAPE is evaluated on model-dependent subsets, making direct comparisons between models potentially misleading.
W2.
The complete product-specific rule inventory and composition formulas are not disclosed, which may limit independent verification and reproducibility.
The expert validation results also require clarification:
- Approximately 6% of atomic-rule judgments disagree with reference labels. It is unclear whether these reflect expert errors, case ambiguity, or reference annotation issues.
- Expert rule accuracy (93.99%) and rule-vector exact match (92.46%) are unusually close. The aggregation method and distribution of errors across cases would help explain this result.
Minor Comments
- The term propagation appears to describe consistency under factual changes rather than sequential reasoning or causal error propagation within models. This distinction should be clarified.
- Uncertainty intervals for the main results would be helpful, potentially using the family-level paired bootstrap already employed in Section 5.5.
- The generator ablation focuses on payout decisions. Examining the stability of PAF and MME across generators would strengthen the evaluation.
Questions for authors
Q1. Are the calculation rules provided to experts also available to models? Can payout errors be separated into eligibility, rule-application, and arithmetic errors?
Q2. Which rule inventories and formulas will be publicly available? How can undisclosed reference answers be verified? Can the authors further analyze expert–reference disagreements and their distribution across cases?
Q3. How does PAF differ between pairs with and ? What proportion of MME-eligible pairs have zero reference payouts at both endpoints?
Q4. Do PAF and MME comparisons remain consistent when using fixed reference-defined subsets or unconditional failure rates?
12 comments
Author response to reviewer 1aNA (Part2, for W2&Q2)
We thank the reviewer for the comments. We will address your concerns in the following aspects. This is Part2. Please see details in Appendix A, D~G
3.Response Regarding Reproducibility and Consistency (W2, Q2)
3.1 Release and verifiability
We have provided the relevant evaluation prompts in Appendix D and presented specific cases together with their corresponding calculation formulas in Appendix E. These materials include case files, atomic rules, contract-file source mappings, expert combination formulas, payout formulas, complete prompts, and an actual response. In Section F, we provide the full texts of real cases from two other insurance types, automobile insurance and property insurance.
Furthermore, We will release a verifiable subset of InsClaimBench together with the complete evaluation and verification pipeline. We will not release the underlying materials obtained from insurance companies, nor cases that involve personal privacy or insurers’ confidential business information. Although all case files are generated, they are grounded in real claim materials provided by insurers, and some retain details traceable to those materials; these cases are excluded from release.
The subset will contain at least 50% of the case families, covering all four product types with complete base–variant structures. For each case, we will release the case file, the contract terms, and reference labels at every level: atomic rules, modules, payout decision, and amount. The pipeline will include all atomic rules and composition formulas used in the subset, the payout formula packages, the prompts for every stage, the full evaluation configuration (model identifiers, call dates, decoding and reasoning settings, retry policy, and parse-failure handling), an executor that recomputes module labels, decisions, and amounts from atomic labels and case facts, and a scorer that computes every metric in the paper. We will also report all main results on the subset, so that third parties can reproduce them exactly and check that the subset is representative of the full benchmark.
With these materials, the reference answers can be verified in three ways. First, all downstream references are deterministic functions of the atomic labels and case facts, so the executor shows directly whether they follow from the atomic labels. Second, each atomic label can be checked against its case file and rule text. Third, for released cases that were part of our expert validation sample, the blinded expert labels provide an independent reference produced without access to our answers.
3.2 Details of Expert Validation
The experts conducted an independent quality assessment after the benchmark had been finalized. Expert feedback was not used to modify the labels. This was an independent validation procedure, and no post hoc label revisions were performed. The sampling design, evaluation units, and inter-expert agreement statistics have already been documented in Appendix A. Our blinded evaluation stratifies case families by insurance type and samples approximately 10% of the families while preserving their complete Base–Variant structures. Four experts independently conducted pairwise evaluations on two non-overlapping subsets, yielding 5,416 and 5,091 pairs of atomic-rule judgments, respectively. As shown in Tables 7 and 8, the raw agreement rates of the two groups are 91.54% and 91.30%, respectively, with Cohen's values of 0.822 and 0.805.
3.3 Expert Rule Accuracy and Rule-Vector Exact Match Are Close
The closeness of these two metrics arises from the correlation and concentration of expert judgment errors across cases. We further explain the underlying mechanism.
Atomic-rule accuracy measures the correctness of individual rule judgments, whereas rule-vector exact match requires all applicable rules within the same case to be judged correctly.
In some cases, if an expert misunderstands a critical fact of a case, multiple rule judgments may be incorrect simultaneously. Conversely, when the expert correctly understands the critical facts, the rule judgments are generally all correct.
For example, in Critical Illness Insurance, an expert needs to determine whether the patient was aware of the disease before the insurance policy was issued or during the waiting period after issuance. If this fact is misunderstood, it may lead to incorrect interpretations of multiple rules, including but not limited to exclusion clauses and liability attribution.
The detailed results are shown below.
| Concentration Metric | Expert A | Expert B | Expert C | Expert D |
|---|---|---|---|---|
| Proportion of cases with errors among all cases | 7.28% | 7.77% | 7.80% | 7.32% |
| Rule error rate within cases containing errors | 78.01% | 72.52% | 76.05% | 78.09% |
| Within-case correlation coefficient between two rule errors, | 0.760 | 0.709 | 0.739 | 0.763 |
Author response to reviewer 1aNA (Part3, for Q3&4)
We thank the reviewer for the comments. We will address your concerns in the following aspects. This is Part3. Please see details in Appendix B.3.
4.Further Evaluation of PAF and MME (Q3 & Q4)
4.1 PAF and MME Results Across Different Subsets (Q3)
We supplement our analysis with metrics calculated on two different sample subsets to address the reviewer's question. We find that even when a model correctly predicts a zero payout amount, its internal module judgments may still contain errors, which can be masked by the final correct zero-payout prediction.
For PAF (Payment Adjustment Failure), we distinguish between pairs in which the reference payout amount should change () and pairs in which the reference payout amount should remain unchanged (). For all six models, the failure rates when the reference payout amount should change (11.40%–38.08%) are higher than those when the reference payout amount should remain unchanged (5.70%–10.65%).
For MME (Masked Module Error), we select case pairs in which the reference payout amount remains unchanged, while the reference label of at least one module is required to change. The event rates range from 22.70% to 61.35%. When we further require the model to correctly predict the payout amounts at both endpoints, zero–zero pairs account for 96.19%–99.14% of the MME-eligible case pairs for each model.
4.2 Robustness Analysis of PAF and MME Using Fixed Denominators (Q4)
As in the second part of our response, we use fixed denominators determined by the reference answers to recalculate PAF and MME. We find that even when using common, model-independent denominators, both failure patterns remain observable. This further supports the necessity of separately evaluating payout amount adjustments under factual changes and intermediate module errors in InsClaimBench.
-
For PAF. We use all case pairs with changes in reference rule judgments as the common denominator for all models and report the proportion in which PAF events occur.
-
For MME. We use case pairs defined by the reference answers as the common denominator for all models and report the proportion in which MME events occur.
Conditional PAF and conditional MME characterize failures among samples satisfying specific prerequisite conditions, whereas fixed-denominator event rates measure the actual frequency of these failure patterns within a common reference-defined sample population.
Therefore, as the reviewer said, the fixed-denominator analysis provides an important complement to the original conditional metrics, allowing us to further distinguish a model's conditional failure tendency from the overall occurrence frequency of failure events.
Under the fixed-denominator setting, the PAF event rates range from 3.60% to 6.76%, while the MME event rates range from 22.70% to 61.35%.
| Model | PAF Amount Changed | PAF Amount Unchanged | Conditional PAF | Fixed-Denominator PAF | Conditional MME | Fixed-Denominator MME |
|---|---|---|---|---|---|---|
| GPT-5.6-Sol | 13.94 | 10.65 | 11.25 | 6.02 | 34.23 | 23.31 |
| Gemini-3.8-Flash | 11.40 | 8.58 | 9.27 | 5.55 | 27.41 | 22.70 |
| DeepSeek-v4.1-Flash | 12.48 | 5.70 | 8.69 | 3.63 | 68.03 | 61.35 |
| Qwen3.8-Flash | 14.91 | 8.31 | 10.81 | 4.64 | 66.96 | 46.01 |
| Kimi-K2.6 | 38.08 | 10.63 | 18.89 | 6.76 | 49.63 | 41.10 |
| Qwen3.5-27B | 28.51 | 8.97 | 14.39 | 3.60 | 61.54 | 44.17 |
Author response to reviewer 1aNA (Part1, for W1&Q1)
We thank the reviewer for the comments. We will address your concerns in the following aspects. This is Part1. Please see details in Appendix B.3.
1. Consistency of Calculation Information Between Models and Experts and Attribution of Payout Errors (W1, Q1)
Human experts and evaluated models receive the same input information. Experts are provided with case-specific public facts, contract terms, and calculation information. They make atomic-rule judgments, assess modules, and select appropriate formulas to calculate payout amounts. The calculation information is derived from insurance contract terms, professional claims-handling rules, and experts' practical experience.
Even with access to the same calculation rules, models may make errors in formula selection, rule application, and parameter extraction. We therefore conducted an error attribution analysis, distinguishing three categories.
Eligibility errors (5.2%). The model incorrectly determines claim eligibility, accounting for the smallest proportion of errors.
Rule-application errors (75.72%). The model selects incorrect payout formulas or substitutes incorrect parameters, such as omitting purchased additional coverage or deducting salvage value twice. This is the largest error category.
Calculation errors (19.08%). The model makes arithmetic mistakes, such as calculating 43,200 + 3,600 = 49,200 instead of 46,800.
These findings indicate that evaluating claims-handling capabilities requires considering eligibility determination, rule application, and payout calculation simultaneously.
2. Evaluation Fairness of Amount Accuracy and MAPE (W1)
We thank the reviewer for pointing out that selecting evaluation samples based on models' own predictions may introduce sample selection effects and affect direct comparisons across models. We therefore introduce two additional metrics for analysis. We find model rankings remain almost unchanged from the main paper, while these metrics reveal additional error patterns.
2.1 Conditional Amount Accuracy
We originally adopted unconditional accuracy, assigning zero to denied claims, because the sets of correctly classified cases differ across models, complicating comparisons. Following the reviewer's suggestion, we additionally evaluate amount accuracy on cases where the model predicts a payable claim and outputs a positive amount.
Conditional Amount Accuracy is the percentage of these cases whose predicted payout matches the reference amount within a tolerance of the larger of 1 monetary unit or 0.01% of the reference payout. The results are 65.27% (GPT-5.6-Sol), 73.91% (Gemini-3.8-Flash), 73.08% (DeepSeek-v4.1-Flash), 61.25% (Qwen3.8-Flash), 54.07% (Kimi-2.6), and 29.31% (Qwen3.5-27B).
2.2 Fixed-Positive MAPE
The reviewer noted that MAPE is calculated on model-dependent subsets, potentially introducing sample selection effects.
We therefore introduce Fixed-positive MAPE, using all cases with positive reference payouts as a common evaluation set. For each case, we calculate the absolute percentage error between the predicted and reference payout amounts and average these errors over the entire fixed set. Incorrect denials, missing payouts, and invalid amount predictions are assigned zero predicted payouts.
The results are 20.70% (GPT-5.6-Sol), 27.91% (Gemini-3.8-Flash), 34.50% (DeepSeek-v4.1-Flash), 29.14% (Qwen3.8-Flash), 35.13% (Kimi-2.6), and 49.12% (Qwen3.5-27B).
These results indicate that Qwen3.5-27B performs well on zero-payout cases, correctly denying many claims and thereby increasing its Acc Joint. Reporting conditional amount accuracy alongside Fixed-positive MAPE helps reveal this error pattern.