- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ROUTE-B2B, a framework for evidence-based business-to-business (B2B) buyer qualification under incomplete information. The framework integrates two components: Partially Identified Evidence Routing (PIER), which optimizes evidence acquisition and stopping decisions using qualification bounds and budget-constrained planning, and Constraint-Compatibility Interface Router (CCIR), which selects extraction interfaces based on calibrated evidence-contract correctness. The authors construct a controlled multilingual benchmark spanning four business domains and five languages, evaluating three open-weight language models and comparing acquisition strategies against several agentic approaches using a hosted model. Experimental results demonstrate that PIER reduces evidence acquisition costs while maintaining or improving observed decision resolution, whereas CCIR substantially improves exact evidence extraction over unconstrained generation, although its performance relative to fixed anchor extraction varies across models.
Critical strengths
S1. The paper addresses an important challenge in evidence-based decision-making: distinguishing missing information from evidence of nonqualification. The explicit treatment of uncertainty and verification requirements provides a principled alternative to conventional scoring approaches.
S2. The qualification bounds, decision rules, and budget-constrained acquisition objective are formally defined. Proposition 1 establishes a conditional soundness guarantee under explicit assumptions about evidence validity and identity verification.
S3. Separating evidence acquisition (PIER) from evidence interpretation (CCIR) is conceptually useful and enables independent evaluation of the two components.
Critical weaknesses
W1. The benchmark consists entirely of programmatically generated fictional organizations and documents. Although this enables controlled experiments, it remains unclear whether the proposed methods generalize to real B2B qualification scenarios involving noisy, contradictory, outdated, or difficult-to-access information.
W2. Table 4 shows that CCIR achieves an average exact-card F1 of 62.96%, compared with 64.17% for fixed Anchor extraction. CCIR also requires more extraction calls (1.267 versus 1.000). Therefore, the experiments do not establish a consistent advantage for the proposed interface router over the strongest fixed-interface baseline.
W3. Despite improvements in acquisition efficiency, the absolute correctly resolved decision rates remain relatively low. For example, CCIR+PIER achieves only 8.33% CRDR in the open-weight evaluation, while PIER achieves 17.50% in the hosted comparison. This raises questions about practical usefulness when most candidates remain unresolved.
Questions for authors
Q1. Source acquisition costs are expressed in designed units rather than measured retrieval expenses, latency, or human verification effort. Consequently, the reported efficiency gains may not directly translate into practical cost savings.
Q2. The ablation results suggest that optional stopping accounts for much of PIER’s acquisition-cost improvement. Full-horizon planning provides only limited additional benefits over one-step planning, making the necessity of the more sophisticated planning mechanism less convincing.
Q3. The framework assumes known predicate weights, decision thresholds, source costs, and outcome distributions. The paper does not establish how reliably these quantities can be estimated in practical deployments. Although the appendix discusses prior sensitivity and implementation assumptions, the generalization of these settings remains uncertain.
Q4. Section 6.1 attributes acquisition efficiency improvements to PIER, whereas Section 7 identifies optional stopping as the clearest source of improvement. Appendix B explicitly acknowledges that GreedyCov does not have the same stopping mechanism. Although the authors provide matched stopping ablations, the main comparison does not isolate the benefit of decision-directed planning from the benefit of stopping. A matched-stopping comparison would make the algorithmic contribution clearer.
Q5. Table 3 shows that Free+PIER achieves higher CRDR (9.79%) than CCIR+PIER (8.33%), despite CCIR’s substantially higher exact-card F1. The authors discuss this distinction in Section 7, but the results leave unclear when exact evidence recovery is necessary for correct qualification and when partial evidence is sufficient.
Q6. Table 4 shows that the Point selector achieves a higher macro F1 (64.54%) than CCIR (62.96%) while requiring fewer calls. Although CCIR incorporates additional statistical calibration and agreement mechanisms, the empirical justification for preferring the complete routing policy over this simpler alternative requires further explanation.
Q7. Table 5 reports that PIER achieves 17.50% CRDR under both common and native readouts, whereas Direct Astra achieves 12.50% under the common readout but 32.50% under its native readout. The paper explains that the readouts use different evidence aggregation and decision mechanisms. Nevertheless, this substantial ranking reversal makes it difficult to establish an overall advantage for PIER over agentic alternatives.
Q8. Proposition 1 assumes that admitted predicate bounds contain their true values and that the identity gate is correct. Appendix B explicitly acknowledges that semantic extraction errors can violate these assumptions. Thus, the guarantee establishes logical correctness under valid evidence rather than end-to-end reliability of the proposed system. The distinction should be emphasized when discussing practical qualification safety.
Q9. Equation 2 requires a valid identity gate for both contact and exclude decisions. However, Appendix A states that identity is fixed as resolved in the controlled benchmark. Consequently, the experiments do not evaluate an important component of the proposed qualification framework.
Q10. The open-weight experiments use simulator-supplied truth-frequency priors, whereas the hosted experiments use a common fixed prior. Although these settings are disclosed, it remains unclear how sensitive PIER’s advantages are to inaccurate priors in realistic environments.
7 comments
How CCIR and PIER Connect Evidence Quality to Justified Decisions (2/2)
W2/Q6: How does CCIR's calibration change what reaches the decision rule? CCIR is the extraction policy in our two-layer system. On Qwen, it chooses Anchor in every language: 88.00 exact F1 versus Point's 80.86 at the same one call per request. On Phi, retaining Free for Japanese/Korean improves F1 from Anchor's 40.16 to 45.55. With Full PIER fixed, CCIR yields 40 correct resolutions against 35 for Anchor and 37 for Point (480 backbone-episodes; zero observed wrong resolutions for all three). These CRDR counts are descriptive (per-backbone Holm p=1). Holding CCIR fixed instead, PIER yields 40 versus GreedyCov's 32 correct, reducing designed cost from 2.294 to 2.001 and extraction calls from 6.64 to 5.57. The first comparison changes interpretation; the second changes acquisition, including stopping. They show how each module contributes to the same qualification decision. Across the three backbones CCIR gains 30.89 pp exact F1 over Free and reduces exact-contract incorrect admissions from 21.40% to 2.53%. Relative to Point, that rate is 2.53% vs 3.42%, at 1.267 vs 1.000 calls/request. Point leads on macro F1, and Anchor's incorrect-admission rate is lower still (1.79%). On Gemma, agreement produces 96.13% precision but only 38.85% recall; the sealed criterion against both fixed interfaces remains unmet. These comparisons locate the benefits and the cost of CCIR's calibration rather than inferring its decision value from a single F1 ranking.
Q5: When is the exact witness necessary? The exact metric tests whether a system can reproduce the designated passage. Eq. 2 instead uses the identity gate and aggregate bounds induced by admitted predicate states. The online check verifies a span against the source, not against the hidden gold boundary. A shorter assertion can therefore supply the needed evidence even when it fails exact-card evaluation; whether it is audit-sufficient depends on whether entity, polarity, and context are preserved.
We rescored the unchanged predictions two ways. “Full assertion” allows omission of a layout prefix but requires the whole factual assertion verbatim, including negation and punctuation. “Substring” requires the correct state and a nonempty literal substring of the same gold witness. All original admission checks are retained.
| Policy | Exact / full assertion / substring F1 | Calls/request |
|---|---|---|
| Free | 32.08 / 62.01 / 63.10 | 1.000 |
| Anchor | 64.17 / 64.17 / 64.17 | 1.000 |
| Point | 64.54 / 66.55 / 66.55 | 1.000 |
| CCIR | 62.96 / 64.16 / 64.16 | 1.267 |
Of 308 admitted Free bulletin cards, 295 preserved both the correct state and the entire factual assertion despite missing the canonical boundary. We then held each policy's actual acquisitions fixed, removed only those state-correct, nonexact cards that retain the full assertion (all surfaces), and reapplied Eq. 2. Free+PIER fell from 47 to 20 correct resolutions out of 480 backbone-episodes; CCIR+PIER stayed at 40. This gold-informed diagnostic shows which evidence mattered on the observed paths, not what a replanning system would do. It explains why document F1 is not a surrogate for decision yield and why literal overlap alone is not a general entailment test.
Q7: What causes the ranking reversal under native readout? Direct produces more correct native labels on this cohort. Direct has 26/80 correct resolutions and one wrong, versus PIER's 14 correct and none wrong, at costs 1.916 vs 2.153. The new native Direct-minus-PIER CRDR difference is +15.0 pp, nominal 95% CI [6.25, 25.00] over 40 paired profiles (exploratory; Holm p=0.04395 across six native contrasts, exact two-sided sign flips). On the same Direct runs, Eq. 2 instead gives 10 correct and zero wrong. Moving from common to native retains 9 correct, turns one correct into an incorrect resolution, and turns 17 verify decisions into correct resolutions; 53 stay verify.
Two comparisons answer your question about the ranked methods. For predicting the native label, Direct's 26 correct resolutions matter. For acting on a documented qualification, a gold-card check on the sources actually bought certifies 14 of PIER's 14 correct resolutions and 9 of Direct's 26. The other 17 are correct predictions; the acquired sources do not establish them under Eq. 2. Separating these outputs is the point of ROUTE-B2B's common readout: it credits a system for prediction and separately tests whether it can justify buyer contact or exclusion from its evidence.
Manuscript revisions: explaining unresolved cases and component value
We have revised the manuscript to connect the component comparisons with your questions about attainable qualification yield and practical resource use. The changes below make the evidence behind our November 8 replies directly inspectable in the paper.
1. W3: Why cases remain unresolved
Appendix I, Figure 10 expands the single-budget analysis into a diagnostic of attainable support.
- Budget and visibility: Panel (a) shows the budget curves for paired rich/sparse views, (b) their paired minimum-cost differences among eventually certifiable firms, and (c) the affordable, over-budget, and intrinsically verify categories.
- Acquisition and interpretation: Panel (d) compares realized decisions with reference-evidence ceilings on the recorded acquisition paths and the ceiling over all affordable paths.
These views locate the remaining gaps in catalog availability, acquisition, and interpretation. Appendix H specifies how the reference-evidence evaluator is constructed.
2. Q2/Q4: What planning adds beyond stopping
Section 6.1 now separates the original whole-policy comparison from purchase ordering under a common STOP gate.
- Shared stopping: Table 23 collects the matched controls.
- Planning depth: Table 3(b) retains the horizon-by-STOP ablation, Figure 2 displays the Full-versus-One cost distribution, and Appendix H traces the single episode where their decisions differ.
The revision makes the observed contribution and additional cost of deeper planning concrete.
3. Q3/Q10: Sensitivity to the acquisition forecast
Appendix H specifies an added five-fold cross-fitted prior estimated from other firms' cached extraction outputs, keeping each held-out firm's two views together. Table 23 places this condition beside the simulator and neutral priors, with separate cost and resolution columns.
The accompanying text reports mixed fold directions and a descriptive aggregate, so readers can assess the estimated-prior condition as a cost-resolution tradeoff.
4. W2/Q5/Q6 and Q1: Component value and resource use
Section 7 connects interface behavior to qualification decisions and their resource requirements.
- Interface consequences (W2/Q5/Q6): The per-backbone overlap of episodes resolved by Free and CCIR is connected to Figure 3(c,d)'s assertion rescoring and fixed-path removal diagnostic. Reading this sequence alongside Table 4 shows when interface changes affect qualification and how Anchor and Point compare.
- Resources per supported outcome (Q1): Figure 4(b) complements these decision analyses with total cohort source spending per supported correct outcome. Section 7 specifies the designed-unit accounting and the target-source prices and latency needed to translate it into deployment costs.
The revised discussion thus links component behavior to supported outcomes and the resources used to obtain them.
PIER: Evidence-Backed Qualification Within a Fixed Budget (1/2)
You identified the key distinction in ROUTE-B2B: missing evidence leaves a qualification unresolved rather than turning it into an exclusion. We tested what that principle means operationally. On the hosted cohort, PIER certifies 14 of the 15 cases that could possibly be certified under budget; even with the same stopping rule it spends less than greedy acquisition. The checks below replay frozen evidence with no new model calls or field labels.
W3: What does 17.50% correct resolution mean? At budget 2.45 we enumerated every affordable source subset for each episode, supplied its true evidence, and applied Eq. 2. This gives a clairvoyant upper bound on certificate-backed decisions, not a ceiling on predicting labels.
In S2, 64/160 views are intrinsically verify even with every fact known; another 62 could resolve with all facts but not with any affordable bundle set. Thus the modest absolute yield has two sources: the decision contract/budget and the system's ability to find and interpret evidence. Hosted PIER found 14 of the 15 attainable cases, each supported by gold evidence in its own acquired set. GreedyCov and common-readout MindSearch also found 14. The smaller models leave a much larger gap. This supports a triage use case under a declared standard of proof, not a forecast of real sales throughput.
Q2/Q4/Q10: Match STOP and vary the priors on the same records. We gave GreedyCov the identical Full PIER stop/continue gate at each state; on continue, only the purchase ranking differs. This is a strong control that uses the full planner for the gate. With CCIR fixed, results are correct resolutions / mean source cost:
The hosted row isolates an ordering benefit: with all 80 decisions unchanged, PIER saves 3.56% over matched GreedyCov (difference -0.079, nominal paired 95% CI [-0.108, -0.049]; 40 profiles). Original S2 gives a different tradeoff: Full PIER resolves eight additional backbone-episodes for 0.037 more cost; the paired quality CI includes zero. Holding S2 outputs fixed and assigning (.4,.4,.2) to every offered predicate, including the distractor, yields 41 correct for Full PIER versus 32 for matched GreedyCov, at lower cost (2.061 vs 2.095). PIER itself costs more than under the original prior (2.001). At short horizon PIER retains 39 correct at cost 1.744 on original S2. Thus the measurable contribution is valuing sources and stopping against a qualification decision; the size and even direction of the cost tradeoff depend on priors and horizon.
W1/Q1/Q3/Q8/Q9: What would deployment require? Weights and cutoffs specify a decision mandate; the deploying organization sets them and encodes noncompensable requirements separately. Costs and priors instead need measurement. Actual hosted logs show 2.50 paid purchases/episode for PIER, 2.89 for original GreedyCov and 2.33 for Direct. Purchases are observable; dollar and labor savings are not. A field test should measure tariffs, retrieval times and verifier minutes, estimate source yields on earlier disjoint records, then evaluate verification burden at several budgets. A 3.56% saving in designed units is not a 3.56% saving in money or time.
The rubric had domain-practitioner input; the evaluated organizations, documents and instance labels are generated, with identity fixed resolved. In a separate public-source feasibility audit, only 7/40 seed candidates passed the high-precision operational-identity gate and none met all pre-cutoff proof requirements. That is evidence of a data-access barrier, not field accuracy or 40 negative buyers. Proposition 1 is conditional on correct identity and sound bounds, not end-to-end reliability. A transfer test should start before identity resolution, retain conflicting, unavailable and expired sources, and score false merges, incorrect exclusions, certificate yield and verifier time. We address the witness, selector and native readout comparisons separately.