- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper represents LLM workflows as stochastic semantic evidence graphs (SSEGs), recording dependencies between retrieval, generation, evaluation, and actions. A standard coupling argument propagates local discrepancies through edge influence bounds to bound terminal discrepancy. The proposed contribution connects this decomposition to review, revalidation, and release decisions. Validation includes 35,000 synthetic cases with known bounds; conflict-path localization on MAGIC; citation analysis and targeted review on ALCE, VitaminC, and ToolSandbox; and a sealed 1,200-case FinGovBench study with transfer to a second model. Reported results include improved localization, reduced review scope, and fewer unsafe releases, alongside null and negative results on additional tasks.
Critical strengths
The VitaminC and ToolSandbox results show that retaining dependencies can reduce the scope of revalidation or post-alarm review while preserving the reported decisions or alarm coverage. This is a useful practical benefit: source updates and detected failures can trigger targeted checks instead of requiring the entire workflow to be reviewed. The paper also reports null and negative results that help delimit these benefits.
Critical weaknesses
-
The practical bounds are not established. Theorem 3.1 requires uniform local-discrepancy and edge-influence bounds. Although introduced alongside the definition of , the inequality for is an assumption, as the following sentence acknowledges. Algorithm 1 delegates construction of the bounds to methods named
SimultaneousBoundandInfluenceBoundwithout specifying a procedure that establishes their validity. The paper does not specify the scoring and release rules sufficiently to reconstruct the real-task experiments: the FinGovBench release map, the flat-UQ rule, and the MAGIC score are not defined, and no values of , , or are reported. Without a justified way to obtain these constants for real workflows, the theorem does not provide a usable certificate for the reported applications. Simulations where the bounds are known by construction cannot resolve this gap. -
The comparisons do not establish the added value of SSEG. FinGovBench includes paired cases in which the same local fault can affect the final action in one case but is blocked in the other. The correct decision differs between these cases, but the flat comparator does not receive the workflow connections needed to distinguish them. SSEG receives this information, so its advantage does not establish whether the proposed uncertainty propagation adds anything beyond access to those connections. A simple graph-aware baseline using the same local measurements, reachability, and hard controls seems to be missing. On MAGIC, SSEG uses both lexical scores and intervention measurements, while the comparator uses only lexical scores. The comparison therefore does not isolate the benefit of graph structure.
-
The code does not reproduce the main decisions. The supplied repository's FinGovBench evaluation recomputes metrics from saved release decisions without regenerating those decisions from the model measurements. Appendix A and the README disclose this limitation, but it leaves the central step from measurements to decisions unverified. In addition, the available archive snapshot is dated after the ICLR deadline, and I cannot establish what code was available at that time. This limits both reconstruction of the reported experiment and verification of the submission's supporting implementation.
Questions for authors
-
How are the bounds required by the theorem obtained and justified in the real-task experiments? Does any reported result use the resulting terminal bound?
-
On FinGovBench, does SSEG outperform a simpler method given the same local measurements and workflow connections? On MAGIC, does the gain remain when the comparator also receives the intervention measurements?
-
Can you provide code that regenerates the FinGovBench release decisions from the model measurements, and identify what code was available at the submission deadline?