- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SEAM combines execution ordering and record/replay (Seal), stochastic delta debugging (Explain), selective disagreement-risk control (Abstain), and deterministic component replacement (Mechanize). The experiments use an image pipeline, a ledger agent loop, a synthetic diagnostic panel, and attribution tasks with known injected causes. The paper separates repeatable execution, offline replay, agreement across fresh runs, and correctness. Its main findings are that ordering can remove client-side variation, repeated calls should not automatically count as independent samples, sequential tests can improve explanation coverage, and replacing one component with a deterministic rule does not always reduce disagreement in the full system.
Critical strengths
-
The statistical analysis is useful. The paper shows how treating repeated calls as independent can give overly confident risk bounds. It also clearly separates pairwise disagreement, disagreement across eight runs, and stability on fresh runs.
-
The controls help explain the gains. The experiments separate ordering from auditing and pruning from sequential testing. In Table 5, sequential testing raises D4 coverage from 0.510 to 0.853 with pruning held fixed.
Critical weaknesses
-
The benefit of the full system is unclear. Ordering alone gives all the bitwise-agreement gains in the image study. The paper does not show whether SEAM reduces debugging cost compared with a simpler combination of logging, replay, and stochastic debugging.
-
The main coverage results do not use the guaranteed procedure. Table 5 allows a fallback after timeout, while Proposition 3 covers strict mode. Strict-mode coverage is lower on difficult cases, and most attribution tests use synthetic faults. The guarantees still need validation on real agent failures.
Questions for authors
-
On the same real fault set and execution budget, how much does SEAM reduce the number of debugging runs, manual checks, or total time compared with ordering + cassette + declaration-aware logging + stochastic ddmin? Could the existing records support a small comparison?
-
For each main attribution experiment, please list the procedure used, whether it allows timeout fallback, whether it runs fresh terminal checks, the value of Ncap, total cost, and answer coverage. Could strict and non-strict versions be compared on the same workload with cost, coverage, and errors shown together?
-
Is there at least one real disagreement from a fresh LLM agent where factor interventions can be repeated, strict mode returns an explanation, and the chosen replacement is then checked on fresh cases or recordings? If not, the paper should make clear that the full workflow has not yet been validated together.
Response to Reviewer cTQ5
Thank you for the detailed and constructive review.
W1 / Q1 (debugging cost against simpler combinations) - We agree. The existing records cover the synthetic part - the fixed-repeat, pruning-off arm of our matched ablation is stochastic ddmin without SEAM's additions. We added the image-pipeline divergence and the new LLM case study, reporting executions, coverage, and errors per configuration. On the hardest separable synthetic tier, recovery rises 0.51 to 0.85 at matched cost from the sequential test alone, pruning adds approximately zero, and strict terminal checks show 0.64 at zero observed errors. Manual checks and wall-clock time are not comparable across these workloads, so we instead report executions and model calls.
W2 / Q2 (strict vs non-strict) - Acknowledged, and the revision now reports strict mode as the primary result. Table 5 contains the updated values and the full factorial, D1-D2, and the image-pipeline row are now located in the SDD-calibration appendix of the revised PDF. All arms share instances and random streams, 300 per tier. Fixed-repeat and sequential are cost-matched at alpha = beta = 0.02, and strict uses the bound-valid alpha = beta = 0.1/(334+0.1), at which the 334-test cap gives a 0.1 error bound (only strict runs fresh terminal checks and enforces Ncap = 334; the fallback arms vote at a threshold at timeout). On D4 the three arms are comparable in cost, so sequential nearly doubling the fixed-repeat coverage (0.51 to 0.85) with no observed errors is a legitimate gain. Strict trades coverage for the bound (0.64) and does not return any wrong answers on any tier. The full per-tier coverage, executions and error rates are also found in Table 5.
Q3 (full workflow on a real disagreement) - Not before the current revision. Explain and Mechanize had been validated on different workloads, and the provider endpoint does not let us repeat interventions on its sampling. We added an open-weights case study (Qwen2.5-14B, sampling on) allowing for repeatable interventions, modeling each decision step as a factor whose intervention is sampled decode against a greedy baseline. Over 100 cases, 10 produced a fresh decision disagreement; 5 saw strict mode return a single-step cause that was confirmed by an intervention (holding that step at the deterministic baseline drops the disagreement rate to 0), and the other 5 saw strict mode refuse (sample cap or an invalid pilot band) rather than guess. Replacing the attributed step with a deterministic rule reduced decision pair-disagreement on held-out cases from 0.35 to 0, at a decision-change cost of about 0.10 against the greedy reference. Thus, the full workflow runs end to end on a real disagreement, and strict mode refuses where separation is insufficient.