acceptodds
Withdrawn from ICLR 2027

CasePath: An Agentic, Process-First Architecture for Determining Evidence Requirements

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model (LLM) agents in rule-governed workflows must choose actions that apply to the current case. In evidence collection, a single fact can make a receipt required, irrelevant or premature. Direct checklist generation leaves the model to judge that fact and choose the action in the same output, even when the rules are in context. We introduce CasePath, a process-first architecture that separates these decisions. The model assesses case conditions and documents; executable process control activates the applicable requirements, derives their evidence needs and holds requests whose conditions remain unresolved. Each request retains its justification. Two synthetic claim studies test this design. On 27 held-out theft-case pairs differing in one fact, CasePath makes 15 wrong checklist changes, versus 27 for direct generation and 41 for generation with the process graph in context. On 108 matched tenancy cases with model judgments fixed, checking inherited branch conditions removes 98% of unnecessary requests and more than halves immediate requests, retaining all 635 valid ones. We release both datasets, structured references, predictions and scorers. Executable process state gives agents a control layer between model judgment and evidence requests.

Primary area:
datasets and benchmarks
Keywords:
agentic AI, process-aware reasoning, evidence acquisition, large language models, structured decision-making, explainable AI, human oversight

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_SBTvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents CASEPATH, a neuro-symbolic framework designed to automate evidence requirement determination in rule-governed workflows such as insurance claims and legal tenancy disputes. CASEPATH separates qualitative reasoning—using large language models (LLMs) to judge document sufficiency and evaluate case conditions—from deterministic control logic that evaluates rule trees to request missing evidence.

Critical strengths

S1. Practical Neuro-Symbolic Decoupling: The framework introduces a clean operational separation between non-deterministic LLM reasoning (for document parsing and qualitative judgments) and deterministic control logic (for evaluating condition trees), providing predictable execution in administrative workflows.

S2. Structured Evidence Routing: The work targets an important problem in legal and administrative automation by structuring document collection into explicit rule evaluations and multi-route evidence dependencies.

Critical weaknesses

W1. Limited Architectural Innovation and Ad-Hoc Process Definition. The separation of LLM qualitative condition judgments from deterministic rule execution is a standard design pattern in neuro-symbolic workflows, classical Decision Model and Notation (DMN), and BPMN engines. The architectural conceptualization lacks a fundamental theoretical or system design innovation. Furthermore, the evidence derivation process is defined ad hoc, alternating between independent rule evaluation in Study A and compiled single-route templates in Study B without a unified formal framework.

W2. Lack of Formal Mathematical Problem Definition. The submission introduces core operational concepts—such as obligations, evidence capabilities, routes, and process scopes—through intuitive examples rather than precise symbolic or graph-theoretic formulations. Without formal mathematical tuples or explicit state-space definitions, the theoretical boundary of the problem remains vague.

Additionally, the paper presents two disconnected problem targets across its evaluations: differential checklist prediction (Δ(D)\Delta(D)) in Study A versus full graph node and state generation in Study B. The framework lacks a unified objective function, such as formalizing evidence collection as an optimization problem that minimizes claimant burden under completeness constraints.

W3. Algorithm 1 is presented as a high-level visual schematic diagram with colored blocks rather than a formal algorithmic specification. It lacks formal input/output data type definitions, variable declarations, loop invariants, and computational complexity bounds required for technical rigor. Moreover, the underlying deterministic algorithm is trivial, reducing to standard 3-valued logical conjunctions (∧3\wedge_3) evaluated over condition trees using basic conditional filtering.

W4. Reliance on Synthetic Datasets and Lack of Expert Validation. The empirical evaluation relies entirely on small, self-authored synthetic datasets: 36 theft case pairs and 150 synthetic tenancy claims. These synthetic setups fail to capture real-world operational challenges such as ambiguous complaints, low-quality document uploads, or adversarial claims.

Crucially, the datasets lack validation by human legal or claims adjustment experts, introducing a significant risk of circular evaluation bias. Because both the ground-truth rules and synthetic claims were generated using LLMs and fixed templates, the evaluation primarily measures how well the system matches its own synthetic assumptions.

W5. Restricted Scope, Weak Baselines, and Extraction Sensitivity. The empirical validation is geographically and domain-restricted to Swiss insurance and tenancy law, leaving its generalizability to non-hierarchical, cross-jurisdictional, or highly cyclic workflows unproven. Furthermore, the baselines are limited to custom, unoptimized prompting strategies, omitting comparisons against established neuro-symbolic agent frameworks, tool-augmented agents (e.g., ReAct), or standard DMN solvers.

Questions for authors

See the critical weaknesses W1--W5.

Reviewer_2KweOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CASEPATH, a framework that combines LLM-based case assessment with executable process control. It determines applicable evidence requirements, identifies missing documents, and retains justifications for document requests. The framework is evaluated through two synthetic studies: Study A examines checklist changes under different case conditions, while Study B evaluates inherited branch-condition checking. The results show fewer incorrect checklist changes in Study A and substantially fewer unnecessary requests in Study B under fixed model judgments.

Critical strengths

The controlled experiments demonstrate fewer incorrect checklist changes in Study A and substantially fewer unnecessary requests in Study B while preserving valid requests, providing empirical support for the proposed approach.

Critical weaknesses
  1. The methodological novelty is insufficiently established. While CASEPATH combines LLM-based case interpretation with executable rules for evidence requests, the paper does not clearly demonstrate what is technically new compared with existing neuro-symbolic and process-aware approaches.
  2. The evaluation provides limited evidence of generalizability. Both studies use synthetic cases and evaluate different parts of the framework, without validation on independent public benchmarks or real-world cases. Moreover, the lack of direct comparisons with existing agent and process-aware methods makes it difficult to assess the broader effectiveness of the proposed approach.
Questions for authors
  1. What is the specific technical novelty of CASEPATH compared with existing neuro-symbolic methods and process-aware agent approaches? In particular, what additional capabilities or benefits does the proposed separation of obligations, facts, evidence capabilities, and routes provide, and how are these supported by the experiments?
  2. How are the LLM-extracted rules and process representations validated for correctness and completeness? Given that most errors in Study A originate from the extracted rules , to what extent can these errors be detected and corrected without manual intervention?
  3. Why are the comparisons limited to custom-designed baselines rather than task-adapted implementations of relevant prior methods? How do the current comparisons establish CASEPATH's effectiveness relative to these methods?
  4. Why do Study A and Study B use different datasets to evaluate checklist changes and inherited branch-condition checking separately, rather than evaluating both aspects on each dataset?
Reviewer_CptPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes CasePath, an architecture for deciding which evidence an agent should request in rule-governed workflows such as insurance claims and tenancy disputes. An LLM judges each case condition as true, false, or unresolved and assesses documents on file, while deterministic code combines each obligation's condition with its enclosing branch conditions, holds unresolved requests, selects evidence routes, and records a justification chain for every request. Two synthetic studies, on theft claims and tenancy disputes, compare this design against direct and process-informed generation and against a fixed-judgment ablation, and the authors claim fewer wrong checklist changes and far fewer unnecessary requests.

Critical strengths

S1. Transparent evaluation practice: Predeclared criteria are reported as failed where they fail, input-free and wrong-condition controls are included, and data, predictions, and scorers are released.

S2. Traceable requests: Every request carries a source-to-document chain, which lets the authors attribute most Study A errors to extracted rules (Table 11).

Critical weaknesses

W1. Study A shows no accuracy benefit. All three predeclared accuracy criteria fail (Table 10); the Holm-corrected p-value against Direct and Graph context is 0.797 on 27 held-out pairs. In Table 4 CasePath is worse than Direct on two of three Stolen-property conditions and worse than both Direct and Graph context on all three Settlement-and-recourse conditions (0.00 / 0.50 / 0.00). The headline "fewer wrong changes" comes mainly from 64 baseline withdrawals; counting wrong additions only, CasePath (15) is level with Direct (16).

W2. Study B's 98% result is largely a property of the design. It compares CasePath with its own ablation (Local scope) at fixed judgments; three-valued conjunction can only hold or drop requests, and the references scope requests by dispute type, so retention of valid requests and removal of cross-type requests follow once the dispute-type judgment is right. 710 of the 793 removals are unscored holds. The predeclared conservative contrasts are all negative (Table 16), and the original evaluator accepted none of the 1,050 runs before scoring was switched post hoc. Table 5 compares only against Local scope; the other baselines appear only in the appendix.

W3. Experimental protocol. A single model (gpt-5.6-terra), a single run per configuration with no repeats or variance, no second model of different strength, and no comparison with any prior system; all baselines are self-made prompts whose text is not given. Appendix F argues that prior systems cannot be run on this task, but only in text (Table 18), without any experiment.

W4. Data validity. Both datasets are synthetic and self-authored with no domain-expert review (App. B.1); Study B's held-out inputs were exposed during product work (App. E.2); Study B's temperature was not set and new inference is non-deterministic.

W5. Method presentation. Algorithm 1 is a workflow illustration, not an algorithm; the formal definitions (three-valued conjunction, route selection) exist only as Eq. 1–3 in Appendix A and are never introduced in the main text, so the main paper does not define what the method computes. Figure 2 is a low-resolution raster with a glyph artifact in "bedroom".

W6. Manuscript format. The main-text related work is two short paragraphs that defer all comparison to the appendix. The bibliography mixes DOIs, bare URLs, "Accessed" dates, and insurer policy PDFs with no consistent format.

Questions for authors
  1. (W1) Given that the predeclared accuracy criteria fail and wrong additions are 15 vs 16 for Direct, what accuracy claim does Study A support?
  2. (W2) Are Study B's references generated by the same compiled templates CasePath executes? What is the LLM's accuracy on the dispute-type judgment itself, and how should the 710 holds be counted?
  3. (W3) How many runs underlie each reported number, and were any configurations repeated? Why were no results obtained with a second model?
  4. (W4) What exactly was exposed during product work, and were any extracted rules or references reviewed by a domain expert?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.