acceptodds
Under review as a conference paper at ICLR 2027

InsClaimBench: Benchmarking Insurance Claim Adjudication Across the Decision Chain

Abstract

Recent advances in reasoning-oriented large language models (LLMs) have motivated increasing evaluation of their ability to perform professional decision tasks. Insurance claim adjudication is one such task, requiring models to connect case evidence, insurance rules, intermediate judgments, and payout calculations across a structured decision process. We introduce InsClaimBench, an end-to-end benchmark for evaluating insurance claim adjudication across the decision chain. Grounded in real claim materials and structured insurance rules, InsClaimBench contains 3,780 cases in 375 case families across auto, property, and health insurance, comprising 86,656 atomic rule judgments. It evaluates each claim from atomic rules through adjudication modules to payout decisions and amounts, with controlled factual variants testing whether required changes are correctly propagated across levels. Evaluation of six LLMs reveals a progressive loss of reliability along the decision chain. Payout-decision accuracy ranges from 74.23–80.19%, while joint decision–amount accuracy drops to 47.54–73.15%. Strong local performance also fails to ensure case-level correctness: atomic-rule accuracy reaches 95.48%, whereas rule-vector exact match peaks at only 36.90%, and the most frequent module errors are not necessarily those most associated with final-decision failure. Under factual changes, these inconsistencies further become propagation failures: module updates are less reliable than rule updates, correct local judgments can still yield incorrect payouts, and correct payouts can conceal intermediate errors. These results show that reliable claim adjudication requires consistent composition and propagation across the decision chain.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Benchmark, Structured Decision-Making, Claim Adjudication, Insurance

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_1aNAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces InsClaimBench, a benchmark for insurance claim adjudication that evaluates a decision chain from atomic rules to modules, payout decisions, and final payout amounts. The benchmark uses expert-defined logic to construct reference answers and LLM-generated claim files, including controlled factual variants. It contains 3,780 cases, 375 case families, and 3,405 base–variant pairs.

Experiments with six LLMs show that relatively high atomic-rule accuracy does not necessarily translate into reliable case-level judgments, payout calculations, or consistency under factual changes.

Critical strengths
  • Deterministic reference payout amounts reveal a substantial gap between payout-decision accuracy (74–80%) and joint decision–amount accuracy (48–73%).
  • The large collection of base–variant pairs enables evaluation of consistency under factual changes through metrics such as PAF and MME.
  • The benchmark covers multiple insurance domains, uses expert-defined decision logic, and includes independent expert validation.
Critical weaknesses

W1.

Section 4.2 describes payout calculation using the claim file and predicted payout decision, while Appendix A states that human experts also receive applicable calculation rules. It is unclear whether models have access to the same information, making the payout evaluation conditions difficult to interpret.

Payout errors may also arise from eligibility judgments, rule application, or arithmetic calculations. Separating these sources would clarify what capabilities are being evaluated.

Two metric-related issues also deserve attention:

  • Joint accuracy combines payout decisions and amount accuracy by assigning zero to predicted denials. Amount accuracy should also be reported for truly payable cases correctly predicted as payable.
  • MAPE is evaluated on model-dependent subsets, making direct comparisons between models potentially misleading.

W2.

The complete product-specific rule inventory and composition formulas are not disclosed, which may limit independent verification and reproducibility.

The expert validation results also require clarification:

  • Approximately 6% of atomic-rule judgments disagree with reference labels. It is unclear whether these reflect expert errors, case ambiguity, or reference annotation issues.
  • Expert rule accuracy (93.99%) and rule-vector exact match (92.46%) are unusually close. The aggregation method and distribution of errors across cases would help explain this result.

Minor Comments

  • The term propagation appears to describe consistency under factual changes rather than sequential reasoning or causal error propagation within models. This distinction should be clarified.
  • Uncertainty intervals for the main results would be helpful, potentially using the family-level paired bootstrap already employed in Section 5.5.
  • The generator ablation focuses on payout decisions. Examining the stability of PAF and MME across generators would strengthen the evaluation.
Questions for authors

Q1. Are the calculation rules provided to experts also available to models? Can payout errors be separated into eligibility, rule-application, and arithmetic errors?

Q2. Which rule inventories and formulas will be publicly available? How can undisclosed reference answers be verified? Can the authors further analyze expert–reference disagreements and their distribution across cases?

Q3. How does PAF differ between pairs with q′≠qq' \neq q and q′=qq'=q? What proportion of MME-eligible pairs have zero reference payouts at both endpoints?

Q4. Do PAF and MME comparisons remain consistent when using fixed reference-defined subsets or unconditional failure rates?

12 comments
AuthorsOpenReview ↗

Author response to reviewer 1aNA (Part1, for W1&Q1)

We thank the reviewer for the comments. We will address your concerns in the following aspects. This is Part1. Please see details in Appendix B.3.

1. Consistency of Calculation Information Between Models and Experts and Attribution of Payout Errors (W1, Q1)

Human experts and evaluated models receive the same input information. Experts are provided with case-specific public facts, contract terms, and calculation information. They make atomic-rule judgments, assess modules, and select appropriate formulas to calculate payout amounts. The calculation information is derived from insurance contract terms, professional claims-handling rules, and experts' practical experience.

Even with access to the same calculation rules, models may make errors in formula selection, rule application, and parameter extraction. We therefore conducted an error attribution analysis, distinguishing three categories.

  • Eligibility errors (5.2%). The model incorrectly determines claim eligibility, accounting for the smallest proportion of errors.

  • Rule-application errors (75.72%). The model selects incorrect payout formulas or substitutes incorrect parameters, such as omitting purchased additional coverage or deducting salvage value twice. This is the largest error category.

  • Calculation errors (19.08%). The model makes arithmetic mistakes, such as calculating 43,200 + 3,600 = 49,200 instead of 46,800.

These findings indicate that evaluating claims-handling capabilities requires considering eligibility determination, rule application, and payout calculation simultaneously.

2. Evaluation Fairness of Amount Accuracy and MAPE (W1)

We thank the reviewer for pointing out that selecting evaluation samples based on models' own predictions may introduce sample selection effects and affect direct comparisons across models. We therefore introduce two additional metrics for analysis. We find model rankings remain almost unchanged from the main paper, while these metrics reveal additional error patterns.

2.1 Conditional Amount Accuracy

We originally adopted unconditional accuracy, assigning zero to denied claims, because the sets of correctly classified cases differ across models, complicating comparisons. Following the reviewer's suggestion, we additionally evaluate amount accuracy on cases where the model predicts a payable claim and outputs a positive amount.

Conditional Amount Accuracy is the percentage of these cases whose predicted payout matches the reference amount within a tolerance of the larger of 1 monetary unit or 0.01% of the reference payout. The results are 65.27% (GPT-5.6-Sol), 73.91% (Gemini-3.8-Flash), 73.08% (DeepSeek-v4.1-Flash), 61.25% (Qwen3.8-Flash), 54.07% (Kimi-2.6), and 29.31% (Qwen3.5-27B).

2.2 Fixed-Positive MAPE

The reviewer noted that MAPE is calculated on model-dependent subsets, potentially introducing sample selection effects.

We therefore introduce Fixed-positive MAPE, using all cases with positive reference payouts as a common evaluation set. For each case, we calculate the absolute percentage error between the predicted and reference payout amounts and average these errors over the entire fixed set. Incorrect denials, missing payouts, and invalid amount predictions are assigned zero predicted payouts.

The results are 20.70% (GPT-5.6-Sol), 27.91% (Gemini-3.8-Flash), 34.50% (DeepSeek-v4.1-Flash), 29.14% (Qwen3.8-Flash), 35.13% (Kimi-2.6), and 49.12% (Qwen3.5-27B).

These results indicate that Qwen3.5-27B performs well on zero-payout cases, correctly denying many claims and thereby increasing its Acc Joint. Reporting conditional amount accuracy alongside Fixed-positive MAPE helps reveal this error pattern.

AuthorsOpenReview ↗

Author response to reviewer 1aNA (Part2, for W2&Q2)

We thank the reviewer for the comments. We will address your concerns in the following aspects. This is Part2. Please see details in Appendix A, D~G

3.Response Regarding Reproducibility and Consistency (W2, Q2)

3.1 Release and verifiability

We have provided the relevant evaluation prompts in Appendix D and presented specific cases together with their corresponding calculation formulas in Appendix E. These materials include case files, atomic rules, contract-file source mappings, expert combination formulas, payout formulas, complete prompts, and an actual response. In Section F, we provide the full texts of real cases from two other insurance types, automobile insurance and property insurance.

Furthermore, We will release a verifiable subset of InsClaimBench together with the complete evaluation and verification pipeline. We will not release the underlying materials obtained from insurance companies, nor cases that involve personal privacy or insurers’ confidential business information. Although all case files are generated, they are grounded in real claim materials provided by insurers, and some retain details traceable to those materials; these cases are excluded from release.

The subset will contain at least 50% of the case families, covering all four product types with complete base–variant structures. For each case, we will release the case file, the contract terms, and reference labels at every level: atomic rules, modules, payout decision, and amount. The pipeline will include all atomic rules and composition formulas used in the subset, the payout formula packages, the prompts for every stage, the full evaluation configuration (model identifiers, call dates, decoding and reasoning settings, retry policy, and parse-failure handling), an executor that recomputes module labels, decisions, and amounts from atomic labels and case facts, and a scorer that computes every metric in the paper. We will also report all main results on the subset, so that third parties can reproduce them exactly and check that the subset is representative of the full benchmark.

With these materials, the reference answers can be verified in three ways. First, all downstream references are deterministic functions of the atomic labels and case facts, so the executor shows directly whether they follow from the atomic labels. Second, each atomic label can be checked against its case file and rule text. Third, for released cases that were part of our expert validation sample, the blinded expert labels provide an independent reference produced without access to our answers.

3.2 Details of Expert Validation

The experts conducted an independent quality assessment after the benchmark had been finalized. Expert feedback was not used to modify the labels. This was an independent validation procedure, and no post hoc label revisions were performed. The sampling design, evaluation units, and inter-expert agreement statistics have already been documented in Appendix A. Our blinded evaluation stratifies case families by insurance type and samples approximately 10% of the families while preserving their complete Base–Variant structures. Four experts independently conducted pairwise evaluations on two non-overlapping subsets, yielding 5,416 and 5,091 pairs of atomic-rule judgments, respectively. As shown in Tables 7 and 8, the raw agreement rates of the two groups are 91.54% and 91.30%, respectively, with Cohen's κ\kappa values of 0.822 and 0.805.

3.3 Expert Rule Accuracy and Rule-Vector Exact Match Are Close

The closeness of these two metrics arises from the correlation and concentration of expert judgment errors across cases. We further explain the underlying mechanism.

Atomic-rule accuracy measures the correctness of individual rule judgments, whereas rule-vector exact match requires all applicable rules within the same case to be judged correctly.

In some cases, if an expert misunderstands a critical fact of a case, multiple rule judgments may be incorrect simultaneously. Conversely, when the expert correctly understands the critical facts, the rule judgments are generally all correct.

For example, in Critical Illness Insurance, an expert needs to determine whether the patient was aware of the disease before the insurance policy was issued or during the waiting period after issuance. If this fact is misunderstood, it may lead to incorrect interpretations of multiple rules, including but not limited to exclusion clauses and liability attribution.

The detailed results are shown below.

Concentration MetricExpert AExpert BExpert CExpert D
Proportion of cases with errors among all cases7.28%7.77%7.80%7.32%
Rule error rate within cases containing errors78.01%72.52%76.05%78.09%
Within-case correlation coefficient between two rule errors, ρ\rho0.7600.7090.7390.763
AuthorsOpenReview ↗

Author response to reviewer 1aNA (Part3, for Q3&4)

We thank the reviewer for the comments. We will address your concerns in the following aspects. This is Part3. Please see details in Appendix B.3.

4.Further Evaluation of PAF and MME (Q3 & Q4)

4.1 PAF and MME Results Across Different Subsets (Q3)

We supplement our analysis with metrics calculated on two different sample subsets to address the reviewer's question. We find that even when a model correctly predicts a zero payout amount, its internal module judgments may still contain errors, which can be masked by the final correct zero-payout prediction.

For PAF (Payment Adjustment Failure), we distinguish between pairs in which the reference payout amount should change (q′≠qq'\neq q) and pairs in which the reference payout amount should remain unchanged (q′=qq'=q). For all six models, the failure rates when the reference payout amount should change (11.40%–38.08%) are higher than those when the reference payout amount should remain unchanged (5.70%–10.65%).

For MME (Masked Module Error), we select case pairs in which the reference payout amount remains unchanged, while the reference label of at least one module is required to change. The event rates range from 22.70% to 61.35%. When we further require the model to correctly predict the payout amounts at both endpoints, zero–zero pairs account for 96.19%–99.14% of the MME-eligible case pairs for each model.

4.2 Robustness Analysis of PAF and MME Using Fixed Denominators (Q4)

As in the second part of our response, we use fixed denominators determined by the reference answers to recalculate PAF and MME. We find that even when using common, model-independent denominators, both failure patterns remain observable. This further supports the necessity of separately evaluating payout amount adjustments under factual changes and intermediate module errors in InsClaimBench.

  • For PAF. We use all case pairs with changes in reference rule judgments as the common denominator for all models and report the proportion in which PAF events occur.

  • For MME. We use case pairs defined by the reference answers as the common denominator for all models and report the proportion in which MME events occur.

Conditional PAF and conditional MME characterize failures among samples satisfying specific prerequisite conditions, whereas fixed-denominator event rates measure the actual frequency of these failure patterns within a common reference-defined sample population.

Therefore, as the reviewer said, the fixed-denominator analysis provides an important complement to the original conditional metrics, allowing us to further distinguish a model's conditional failure tendency from the overall occurrence frequency of failure events.

Under the fixed-denominator setting, the PAF event rates range from 3.60% to 6.76%, while the MME event rates range from 22.70% to 61.35%.

ModelPAF Amount ChangedPAF Amount UnchangedConditional PAFFixed-Denominator PAFConditional MMEFixed-Denominator MME
GPT-5.6-Sol13.9410.6511.256.0234.2323.31
Gemini-3.8-Flash11.408.589.275.5527.4122.70
DeepSeek-v4.1-Flash12.485.708.693.6368.0361.35
Qwen3.8-Flash14.918.3110.814.6466.9646.01
Kimi-K2.638.0810.6318.896.7649.6341.10
Qwen3.5-27B28.518.9714.393.6061.5444.17
Reviewer_bdPBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the InsClaimBench benchmark for insurance claims assessment, comprising 3,780 synthetic cases, 375 case families, and 86,656 atomic rule judgments, covering four types of insurance business. The authors evaluate six models from four levels: rule, module, payout decision, and amount, and analyze the consistency of judgments using factual variations of the same case. The research aims to reveal that even if a model correctly judges some rules, it may not necessarily obtain the correct claims outcome.

Critical strengths

Jointly evaluating payout decisions with payout amounts is practically significant, distinguishing between "whether the payout was correct" and "the final payment result was correct."

The case family design further allows for examining which judgments should change and which should remain unchanged after factual changes, providing a more nuanced diagnosis than single-case accuracy.

The paper presents examples of independent expert review, generator replacement experiments, and stepwise calculations for claims. The generator experiment uses paired bootstrap at the case family level, considering the correlation of samples within the same family. Payout tolerance, missing labels, and the denominator for some conditional indicators are also clearly explained.

Critical weaknesses
  1. Experiments cannot yet distinguish between cross-request inconsistency and error propagation in chained reasoning.

Page 6, §4.2 explains that atomic rule judgments receive the complete case and target rule text; module judgments and final decisions are completed through separate requests, not receiving the preceding atomic rule text or predictions; amount judgments receive the case and the predicted payout decision.

Therefore, atomic judgments are not actually passed to modules and the final decision, and the visible information differs across request layers.

This protocol can measure the consistency of the same model under different prompts, but it is insufficient to pinpoint "correct intermediate judgments failing to propagate to subsequent decisions." Information discrepancies may also explain some performance degradation if the complete case does not contain all product terms and calculation rules. The paper needs to provide side-by-side examples of inputs at each layer and confirm which rules are visible to all levels.

It is recommended to compare chained reasoning with independent judgments, explicit propagation of intermediate predictions, and a control group that replaces intermediate judgments with reference answers under the same information conditions. A baseline of "model predicting atomic labels, then calculating the result by a deterministic executor" can also be added. These experiments can further distinguish between errors caused by rule understanding, logical aggregation, and amount calculation.

  1. Core Reference Answers and Evaluation Process Difficult to Verify

Appendix D.1 (page 15) states that the complete product rules, source mapping, and expert combination formulas are not publicly available. The given medical insurance example illustrates the calculation process but is insufficient to verify the reference answer for the entire benchmark. The paper also lacks complete hints, decoding configuration, model version, handling of parsing failures, and a runnable scorer.

The approximately 94% atomic agreement rate between experts and reference labels supports data quality, but it is still necessary to explain how disagreements are arbitrated and which labels were revised as a result. Appendix A (page 13) states that experts can view the original rules and applicable calculation rules; it is necessary to confirm whether the model obtains the same information in each level of judgment and report human-machine comparisons on the same verification subset.

It is recommended to provide auditable samples, rules, and executors covering all insurance types, as well as a complete evaluation configuration. If restricted rules cannot be publicly disclosed, the access method for reviewing or hosting the evaluation should be explained. As a benchmark paper, the auditability of the reference answer and the reproducibility of subsequent research are important prerequisites for evaluating its contribution.

Questions for authors
  1. Can all requests at each level see the complete terms and calculation rules? After ensuring consistent input, does the difference between independent judgment and explicit chained reasoning still exist?

  2. How are disagreements between expert and reference labels handled? How will the restricted rules be made available for review and reproduction?

  3. After adding family-level confidence intervals and hierarchical rule complexity, are the conclusions regarding propagation failure, model ranking, and generator robustness maintained?

6 comments
AuthorsOpenReview ↗

Author response to reviewer bdPB (Part1, for W1&Q1)

We thank the reviewer for the comments. We will address your concerns in the following aspects. This is Part1.

1. Stage-wise inputs, cross-level consistency, and error propagation (W1, Q1)

We thank the reviewers for this important point. An error in a downstream judgment (module, decision, or amount) can relate to the model’s rule-level judgments in two ways. In error propagation, the model’s rule-level judgment is wrong and leads to a wrong downstream judgment. In cross-level inconsistency, the model’s downstream judgment does not agree with what its own rule-level judgments imply. Because our main protocol queries each level independently, our cross-level metrics capture the combined effect of both mechanisms, yet our paper attributed this effect to propagation alone. We agree that the independent protocol cannot separate the two. In addition, stages in the main protocol see different information (only rule requests include rule texts), which could itself produce cross-level differences.

1.1 What each stage sees.

StageCase fileContractAtomic rule textsFormula packageUpstream outputs
Rule (main)✓✓target rule only––
Module (main)✓✓–––
Decision (main)✓✓–––
Amount (main)✓✓–✓own predicted decision
Human experts✓✓all✓own judgments
Matched / Predicted / Oracle (new)✓✓all✓none / own / reference rule labels

In the main protocol, module and decision requests do not see rule texts. Human experts saw all rule texts and the formula package. This information gap may contribute to cross-level differences. We chose independent queries deliberately, so that each level is evaluated directly from the claim file and upstream outputs cannot anchor later judgments. The comparison below removes the information gap.

1.2 On a stratified sample of 805 cases, we compare the main protocol with three conditions in which every stage receives all rule texts and the formula package:

  • Matched Independent: no stage sees upstream outputs.
  • Predicted Chain: each stage receives the model’s own upstream judgments.
  • Oracle Chain: the model’s rule judgments are replaced by reference labels, and the rest of the chain is still produced by the model.

Adjacent conditions differ in exactly one factor, so each difference isolates one effect. We report joint decision–amount accuracy (whether the outcome is correct) and full-chain accuracy (which additionally requires all modules and the decision to be correct).

ModelMainMatchedPredictedOracle
Gemini-3.8-Flash73.29 / 41.2472.17 / 35.0373.91 / 42.3675.40 / 45.96
GPT-5.6-Sol70.31 / 50.3168.82 / 49.9474.29 / 63.6077.76 / 67.58
DeepSeek-V4.1-Flash69.07 / 32.4268.07 / 31.9373.29 / 46.0977.02 / 53.17

Entries are joint / full-chain accuracy (%).

  • The information gap does not explain the failures. Providing full rule texts and formulas does not improve joint accuracy (Matched − Main: −1.0 to −1.5 points), confirming that main-protocol failures are not artifacts of information differences.

  • Both mechanisms are present, mainly in intermediate judgments. Predicted − Matched measures inconsistency, while Oracle − Predicted measures error propagation. Gains are modest in joint accuracy (+1.7–5.5 and +1.5–3.7 points) but substantial in full-chain accuracy (+7.3–14.2 and +3.6–7.1 points). Models thus fail to carry even correct rule judgments into higher-level decisions. For decision-critical rule errors (n = 45 / 48 / 71), Oracle raises decision accuracy by 37.78 / 68.75 / 56.34 points for Gemini / GPT / DeepSeek.

  • Correct outcomes still conceal intermediate errors. Under Matched Independent, 19–37% of cases have correct outcomes despite incorrect module judgments, confirming that correct payouts can mask intermediate errors even under identical information.

  • Composition is the main bottleneck. Even with all rule labels correct, joint accuracy reaches only 75.40–77.76%, leaving 71–88% of Matched errors unresolved. Experts achieve 96.79% decision accuracy with the same information. Most errors therefore arise from composing rule judgments into decisions and amounts, demonstrating the necessity of chain-level evaluation.

AuthorsOpenReview ↗

Author response to reviewer bdPB (Part2, for W2&Q2)

We thank the reviewer for the comments. We will address your concerns in the following aspects. This is Part2.

2.Insufficient Reliability of Reference Answers and Evaluation Reproducibility (W2, Q2)

2.1 Details of expert validation

The expert evaluation was conducted as an independent quality assessment after the Benchmark had been finalized. Expert feedback was not used to modify the labels. This was an independent validation procedure, and no post-hoc label revisions were performed (for Q2). The sampling design, evaluation units, and inter-expert agreement statistics have already been documented in Appendix A. Our blinded evaluation stratifies case families by insurance type and samples approximately 10% of the families while preserving their complete Base–Variant structures.

Four experts independently conducted pairwise evaluations on two non-overlapping subsets, yielding 5,416 and 5,091 pairs of atomic-rule judgments, respectively. As shown in Tables 7 and 8, the raw agreement rates for the two groups were 91.54% and 91.30%, respectively, with Cohen's κ values of 0.822 and 0.805.

2.2 Reproducibility

We have provided the relevant evaluation prompts in Appendix D and presented specific cases and their corresponding calculation formulas in Appendix E. These materials include case files, atomic rules, contract file source mappings, expert combination formulas, payout formulas, complete prompts, and an actual model response. In Appendix F of the main text, we provide the full case texts of real cases from two other insurance types, auto insurance and property insurance.

We will release a verifiable subset of InsClaimBench together with the complete evaluation and verification pipeline. We will not release the underlying materials obtained from insurance companies, nor cases that involve personal privacy or insurers’ confidential business information. Although all case files are generated, they are grounded in real claim materials provided by insurers, and some retain details traceable to those materials; these cases are excluded from release.

The subset will contain at least 50% of the case families, covering all four product types with complete base–variant structures. For each case, we will release the case file, the contract terms, and reference labels at every level: atomic rules, modules, payout decision, and amount. The pipeline will include all atomic rules and composition formulas used in the subset, the payout formula packages, the prompts for every stage, the full evaluation configuration (model identifiers, call dates, decoding and reasoning settings, retry policy, and parse-failure handling), an executor that recomputes module labels, decisions, and amounts from atomic labels and case facts, and a scorer that computes every metric in the paper. We will also report all main results on the subset, so that third parties can reproduce them exactly and check that the subset is representative of the full benchmark.

With these materials, the reference answers can be verified in three ways. First, all downstream references are deterministic functions of the atomic labels and case facts, so the executor shows directly whether they follow from the atomic labels. Second, each atomic label can be checked against its case file and rule text. Third, for released cases that were part of our expert validation sample, the blinded expert labels provide an independent reference produced without access to our answers.

AuthorsOpenReview ↗

Author response to reviewer bdPB (Part3, for Q3)

We thank the reviewer for the comments. We will address your concerns in the following aspects. This is Part3.

3.Statistical Confidence, Rule Complexity, and Robustness of Conclusions (Q3)

We conducted the following supplementary experiments, with detailed results reported in Appendices B.3 and B.4.

The supplementary experiments show that

  • after accounting for within-family correlations, the point-estimate ranking of overall model accuracy remains unchanged, and the failure phenomena reflected by PAF and MME remain observable;
  • there is no consistent relationship between the number of rules and model performance; (for rule complexity)
  • the generator exhibits robustness.

3.1 Family-Level Confidence Intervals and Model Rankings

Based on all 3,780 cases and 375 case families, we performed 10,000 family-level Bootstrap resamples to account for the effects of within-family case correlations on statistical uncertainty.

Furthermore, we calculated confidence intervals for paired differences between models on the same cases and constructed simultaneous 95% confidence intervals for all 15 pairwise model comparisons to account for multiple comparisons.

3.2 Complexity-Stratified Analysis Based on the Number of Atomic Rules

To examine the effect of rule complexity on model performance, we use the number of atomic rules involved in each case as an operational measure of complexity. The results show heterogeneous associations between rule complexity and model performance, indicating that the number of atomic rules alone is insufficient to fully explain performance differences across models.

3.3 Robustness of Cross-Level Inconsistency and PAF/MME

We further performed family-level Bootstrap analyses for the PAF and MME metrics to examine whether the corresponding failure phenomena depend on repeated or correlated samples within case families. Across the six models, PAF incidence rates range from 8.69% to 18.89%, while MME incidence rates range from 27.41% to 68.03%. For all models, the lower bounds of the corresponding family-level 95% confidence intervals are greater than zero. These results indicate that the corresponding failure phenomena remain observable after accounting for correlations within case families, and their existence cannot be attributed solely to a small number of correlated cases.

3.4 Generator Robustness

In the supplementary PAF/MME difference analysis, the confidence intervals for the differences include zero for most models, except Kimi and Qwen3.5-27B, indicating that the corresponding comparisons do not yet show clear differences. Therefore, the results support relative stability in most comparisons, while also revealing the sensitivity of certain models and metrics to changes in generation settings.

Reviewer_Uq9sOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces InsClaimBench, a benchmark for evaluating LLMs on insurance claim adjudication, covering atomic rule judgments, adjudication modules, payout decisions, and amount calculations. The benchmark contains 3,780 cases and introduces counterfactual variants to evaluate models' responses to factual changes. Experiments on six LLMs show that high local judgment accuracy does not necessarily translate into correct final decisions, and models exhibit substantial inconsistencies across decision levels and in payout adjustments.

Critical strengths
  1. The paper decomposes insurance claim adjudication into atomic rules, adjudication modules, payout decisions, and amount calculations, providing more fine-grained diagnostic capabilities than evaluating final answers alone.

  2. By introducing controlled factual changes, the benchmark evaluates whether models correctly update relevant judgments, preserve unaffected judgments, and adjust final payout amounts accordingly.

Critical weaknesses
  1. The paper evaluates rules, modules, and final decisions through independent queries without passing upstream predictions to downstream stages. Therefore, the observed results primarily reflect inconsistencies across decision levels rather than demonstrating actual error propagation through a sequential decision process. This weakens the paper's central claims about propagation failures.

  2. The paper does not provide the complete expert-defined rules and composition formulas. Although human validation is conducted, the lack of these details makes it difficult for third parties to independently verify the reference answers and limits the reproducibility of the benchmark.

Questions for authors

Q1. Since different decision levels are evaluated independently, how do the authors distinguish cross-level inconsistency from actual error propagation? Have the authors considered evaluating a sequential pipeline where upstream predictions are explicitly passed to downstream stages?

Q2. Do the authors plan to release the complete atomic rules, composition formulas, reference labels, and evaluation scripts? If some components cannot be released, how can third parties independently verify the benchmark?

Q3. Compared with existing insurance reasoning benchmarks such as InsLogicBench, what are the main additional evaluation capabilities introduced by InsClaimBench?

5 comments
AuthorsOpenReview ↗

Author response to reviewer Uq9s (Part1, for W1&Q1)

We thank the reviewer for the comments. We will address your concerns in the following aspects. This is Part1.

1.How to Distinguish Cross-Request Inconsistency and Error Propagation in Chained Reasoning (W1&Q1)

We thank the reviewers for this important point. Downstream errors can arise from error propagation, where incorrect rule judgments lead to incorrect downstream judgments, or cross-level inconsistency, where downstream judgments contradict the model’s own rule-level judgments. Our independently queried main protocol captures both effects without separating them, although the paper attributed them to propagation alone. Differences in information across stages may also contribute.

We therefore re-evaluated three models on 805 cases under identical information, comparing independent and chained protocols. The results establish four findings.

(1) Information differences do not explain cross-level failures, as equalizing inputs does not improve accuracy.

(2) Both inconsistency and propagation exist, with modest effects on final outcomes but substantially larger effects on intermediate judgments.

(3) Correct outcomes frequently mask intermediate errors, even under identical information.

(4) Composition remains the main bottleneck. Most errors persist even with reference rule labels, demonstrating that the central challenge lies in composing rule judgments into claim decisions and payout amounts, precisely the capability targeted by chain-level evaluation.

1.1 What each stage sees.

StageCase fileContractAtomic rule textsFormula packageUpstream outputs
Rule (main)✓✓target rule only––
Module (main)✓✓–––
Decision (main)✓✓–––
Amount (main)✓✓–✓own predicted decision
Human experts✓✓all✓own judgments
Matched / Predicted / Oracle (new)✓✓all✓none / own / reference rule labels

In the main protocol, module and decision requests do not see rule texts. Human experts saw all rule texts and the formula package. This information gap may contribute to cross-level differences. We chose independent queries deliberately, so that each level is evaluated directly from the claim file and upstream outputs cannot anchor later judgments. The comparison below removes the information gap.

1.2 On a stratified sample of 805 cases, we compare the main protocol with three conditions in which every stage receives all rule texts and the formula package:

  • Matched Independent: no stage sees upstream outputs.
  • Predicted Chain: each stage receives the model’s own upstream judgments.
  • Oracle Chain: the model’s rule judgments are replaced by reference labels, and the rest of the chain is still produced by the model.

Adjacent conditions differ in exactly one factor, so each difference isolates one effect. We report joint decision–amount accuracy (whether the outcome is correct) and full-chain accuracy (which additionally requires all modules and the decision to be correct).

ModelMainMatchedPredictedOracle
Gemini-3.8-Flash73.29 / 41.2472.17 / 35.0373.91 / 42.3675.40 / 45.96
GPT-5.6-Sol70.31 / 50.3168.82 / 49.9474.29 / 63.6077.76 / 67.58
DeepSeek-V4.1-Flash69.07 / 32.4268.07 / 31.9373.29 / 46.0977.02 / 53.17

Entries are joint / full-chain accuracy (%).

  • The information gap does not explain the failures. Providing full rule texts and formulas does not improve joint accuracy (Matched − Main: −1.0 to −1.5 points), confirming that main-protocol failures are not artifacts of information differences.

  • Both mechanisms are present, mainly in intermediate judgments. Predicted − Matched measures inconsistency, while Oracle − Predicted measures error propagation. Gains are modest in joint accuracy (+1.7–5.5 and +1.5–3.7 points) but substantial in full-chain accuracy (+7.3–14.2 and +3.6–7.1 points). Models thus fail to carry even correct rule judgments into higher-level decisions. For decision-critical rule errors (n = 45 / 48 / 71), Oracle raises decision accuracy by 37.78 / 68.75 / 56.34 points for Gemini / GPT / DeepSeek.

  • Correct outcomes still conceal intermediate errors. Under Matched Independent, 19–37% of cases have correct outcomes despite incorrect module judgments, confirming that correct payouts can mask intermediate errors even under identical information.

  • Composition is the main bottleneck. Even with all rule labels correct, joint accuracy reaches only 75.40–77.76%, leaving 71–88% of Matched errors unresolved. Experts achieve 96.79% decision accuracy with the same information. Most errors therefore arise from composing rule judgments into decisions and amounts, demonstrating the necessity of chain-level evaluation.

AuthorsOpenReview ↗

Author response to reviewer Uq9s (Part2, for W2&Q2)

We thank the reviewer for the comments. We will address your concerns in the following aspects. This is Part2.

2. Release and verifiability (W2&Q2)

For Q2, yes. We have updated the manuscript and provided a complete reproducible example in Appendices D and E, including the case file, atomic rules, contract files and source mappings, expert combination formulas, payout formulas, complete prompts, and an actual model response. Appendix F further provides original case texts from real automobile and property insurance claims. These materials enable reproduction of the audit results and the full workflow.

We will release a verifiable subset covering at least 50% of InsClaimBench's case families, spanning all four insurance types with complete base–variant structures. The underlying insurer-provided materials and cases containing personal or commercially confidential information, including traceable details retained in some generated cases, will be excluded.

For each released case, we will provide the case file, contract terms, and reference labels for atomic rules, modules, payout decisions, and amounts. The complete evaluation and verification pipeline will include all relevant atomic rules, composition and payout formulas, stage-specific prompts, evaluation configurations (model identifiers, call dates, decoding and reasoning settings, retry policies, and parse-failure handling), a deterministic executor for recomputing downstream labels and amounts, and a scorer implementing every reported metric. We will also report all main results on this subset, enabling exact reproduction and assessment of its representativeness.

The reference answers can be independently verified at three levels. Downstream labels and amounts can be recomputed deterministically from atomic labels and case facts; atomic labels can be checked against case files and rule texts; and released cases included in expert validation can be cross-checked against blinded expert judgments obtained without access to our reference answers.

AuthorsOpenReview ↗

Author response to reviewer Uq9s (Part3, for Q3)

We thank the reviewer for the comments. We will address your concerns in the following aspects. This is Part3.

3. Differences from InsLogicBench (Q3)

We thank the reviewer for specifically highlighting InsLogicBench. We agree that it is a closely related benchmark and appreciate the opportunity to clarify the additional evaluation capabilities introduced by InsClaimBench.

InsLogicBench focuses on policy-clause-based reasoning and the quality of argumentation. However, InsLogicBench has difficulty addressing complex real-world scenarios and other forms of potential noise. Building on this, InsClaimBench evaluates the consistency of interconnected adjudicative judgments, their responses to controlled factual changes, and their financial consequences. Specifically, InsClaimBench extends the evaluation scope along three complementary dimensions.

Cross-Level Adjudication Consistency. InsClaimBench establishes an expert-defined decision structure linking atomic-rule judgments, three adjudication modules (contract and claim eligibility, event and causation, and coverage and exclusions), final claim decisions, and payout amounts. This enables independent evaluation at each level and systematic examination of inconsistencies between local judgments and case-level outcomes, including cases where all atomic judgments are correct but the final decision is incorrect, or vice versa.

Financial Outcome Evaluation. Beyond binary claim decisions, InsClaimBench also evaluates contract-determined payout amounts and the joint correctness of payout decisions and amounts. This enables the identification of errors with significant financial consequences that cannot be detected through decision accuracy alone.

Counterfactual Decision Consistency and Consequence Propagation. InsClaimBench introduces controlled Base–Variant pairs in which selected claim facts are modified while irrelevant evidence and applicable rules remain unchanged. These pairs enable the evaluation of whether models correctly update affected judgments, preserve unaffected judgments, and produce appropriate downstream payout outcomes. We further quantify module update failures, payment adjustment failures, and intermediate judgment errors masked by correct payout outcomes. For example, a factual change may not affect a claim's payment eligibility but may require a different payout amount, allowing us to examine whether the financial consequences are correctly reflected.

These additional capabilities demonstrate that strong local reasoning performance does not necessarily translate into reliable end-to-end claims adjudication or consistent responses to factual changes.

Reviewer_JPCKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces InsClaimBench, a novel benchmark designed to evaluate large language models (LLMs) across the end-to-end insurance claim adjudication process. The benchmark comprises 3,780 cases spanning auto, property, and health insurance, covering the complete decision chain from fundamental policy rule interpretation to final payout calculation.

Critical strengths
  1. Rigorous evaluation through counterfactual testing. The benchmark incorporates carefully controlled counterfactual pairs to examine whether models can appropriately update their decisions in response to changes in critical evidence.
  2. Insightful analysis of reasoning failures. The experimental findings reveal important limitations of current LLMs in complex decision-making tasks, including cases where correct final outcomes mask intermediate reasoning errors, which provides valuable insights beyond conventional accuracy-based evaluation.
Critical weaknesses

Limited representation of real-world complexity. The benchmark relies primarily on synthetic cases, which may not adequately capture the missing evidence, conflicting information, ambiguous documentation, and other forms of noise commonly encountered in real-world insurance claims.

Questions for authors

Given the reliance on synthetic data, do the authors have plans or preliminary results involving real-world, potentially noisy and unstructured claim documents?

4 comments
AuthorsOpenReview ↗

Follow-up on Reviewer JPCK Concerns and Additional Clarification

We sincerely thank you again for your time and constructive feedback.

We would also like to highlight that Appendix E.1 in the new version presents a de-identified case file derived from an actual insurance claim provided by one of our collaborating insurance companies. This example further illustrates the real-world claims materials underlying the construction of InsClaimBench and complements our earlier clarification regarding the benchmark's real-world grounding.

As the discussion period is approaching its end, we would like to kindly follow up and ask whether our responses have adequately addressed your initial concerns. If any point would benefit from further clarification, we would be glad to provide additional details.

Thank you again for the time and effort you have devoted to reviewing our work.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you again for raising this important concern. In response, we have supplemented the manuscript with additional real-world claim materials to provide further evidence of the benchmark's connection to practical insurance claims handling.

In particular, Appendix E.1 now includes a real claim file provided by an insurance company after de-identification. We have also added the original texts of real claims from automobile and property insurance in Appendix F. These examples provide additional context on how real-world claim materials are represented and processed in our evaluation.

As the discussion period is approaching its end, we would like to kindly follow up and ask whether these additional materials adequately address your concerns regarding the reliance on synthetic cases and the representation of real-world claim complexity. We would greatly appreciate any further feedback or clarification requests. Thank you again for your time and consideration.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you again for raising this important concern. In response, we have supplemented the manuscript with additional real-world claim materials to provide further evidence of the benchmark's connection to practical insurance claims handling.

In particular, Appendix E.1 now includes a real claim file provided by an insurance company after de-identification. We have also added the original texts of real claims from automobile and property insurance in Appendix F. These examples provide additional context on how real-world claim materials are represented and processed in our evaluation.

As the discussion period is approaching its end, we would like to kindly follow up and ask whether these additional materials adequately address your concerns regarding the reliance on synthetic cases and the representation of real-world claim complexity. We would greatly appreciate any further feedback or clarification requests. Thank you again for your time and consideration.

AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer,

We have carefully addressed your concerns and incorporated the corresponding revisions into our manuscript. As the discussion period is approaching its end, we would be happy to provide any further clarifications if needed.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

InsClaimBench — Rebuttal Summary

Core Contributions

We introduce InsClaimBench, a benchmark for evaluating the complete insurance claim adjudication chain. It contains 375 case families, 3,780 cases, and 86,656 atomic-rule judgments, covering atomic rules, module judgments, claim decisions, and payout amounts.

Our work extends rule-level reasoning evaluation to cross-level decision consistency, payout correctness, and responses to factual changes. Our experiments show that correct local judgments may lead to incorrect final decisions, while correct payouts may hide intermediate errors. These findings highlight the importance of evaluating the complete decision chain in complex professional tasks.

Main Additions During the Rebuttal

Sequential Decision Experiments and Error Propagation (Reviewers Uq9s & bdPB)

We conducted additional sequential decision experiments on 805 stratified cases. We compared matched independent evaluation, predicted chains, and Oracle chains with the original setting. The results show that information differences do not explain the main failures. Passing upstream predictions and correcting atomic-rule errors improve full-chain accuracy. For cases with decision-critical rule errors, Oracle intervention improves decision accuracy by 37.78–68.75 percentage points, while many errors remain even with correct atomic-rule labels. These results confirm that rule composition and cross-level reasoning remain key challenges.

Statistical and Evaluation Robustness (Reviewers bdPB & 1aNA)

We strengthened the statistical tests and evaluation metrics. We added confidence intervals, paired model comparisons, multiple-comparison adjustments, rule complexity analysis, generator robustness tests, and further PAF/MME analyses. We also refined the amount accuracy and MAPE evaluation and introduced Revised Amount Accuracy to include zero, missing, and invalid predictions. The overall model ranking remains unchanged after accounting for within-family correlations, and the PAF/MME failure patterns remain observable. These results strengthen the reliability and robustness of our main findings.

Real-World Complexity Analysis (Reviewer JPCK)

We added an analysis of real-world claims complexity. We clarified how InsClaimBench was developed from real insurance claims, contract terms, and expert knowledge. We also evaluated 975 cases with missing evidence and 2,552 cases with conflicting facts or evidence across six models. The results show clear gaps between atomic-rule accuracy and module-level performance. These findings further support the practical relevance of InsClaimBench.

Data Release and Reproducibility (Reviewers Uq9s & bdPB)

We expanded the reproducibility materials and clarified our data release plan. We added complete evaluation prompts, case materials, contract rules, composition and payout formulas, and actual model responses. We also committed to releasing a verifiable subset covering at least 50% of case families, including all four insurance types and complete Base–Variant structures, subject to privacy and confidentiality requirements. The release will include reference labels, evaluation settings, deterministic computation tools, and scoring code. These additions improve the transparency, verifiability, and reproducibility of InsClaimBench.

Overall Summary

We updated the manuscript based on the reviewers' feedback and our additional analyses. Overall, our rebuttal strengthens InsClaimBench in error mechanism analysis, evaluation robustness, real-world relevance, and reproducibility.

We sincerely thank all reviewers, Area Chairs, Program Chairs, and members of the ICLR organizing committee for their time, dedication, and efforts throughout the review process. We greatly appreciate their constructive feedback and contributions to the research community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.