- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies repairing an LLM workflow after part of it has already executed. ObliGate records artifacts and their dependencies, uses runtime evidence to restrict possible interventions, probes candidate repair roots on cloned states, and replays the affected downstream computation. A repaired state is adopted if its score under the fixed verifier improves. The evaluation includes strict MBPP+, injected failures, and runtime-flagged checkpoints, as well as code, mathematics, and QA workflow comparisons. On strict MBPP+, the full method reports 80.95% raw pass@1 at 325 task-action tokens under the 1,800-token ceiling, with lower token use than fixed deep execution.
Critical strengths
-
Avoiding a full restart after a localized failure is a useful objective. The dependency-closed replay mechanism gives a concrete way to reuse valid work while keeping the affected artifacts consistent. This is a more careful treatment of repair execution than simply retrying the final answer.
-
The paper includes feedback-based repair controls and distinguishes controlled faults from naturally flagged cases. These comparisons allow readers to separate the benefit of ordinary draft correction from that of the broader repair mechanism.
Critical weaknesses
-
The full method's added value is unclear in the natural-failure study. Full ObliGate and closed-draft replay both improve 76 of 228 cases, while their reported end-to-end token costs are approximately 19,549 and 7,935, respectively. The full method has a lower false-repair rate, but probing adds no aggregate recoveries. All improvements are draft-local. None of the 37 upstream-involving cases improves. These results support local draft replay more directly than they support the broader cross-artifact repair proposal.
-
The other results offer limited evidence for the full method's advantage. On strict MBPP+ at the 1,800-token ceiling, Trace-Repair reaches 79.98% at 290 task-action tokens, close to the full method's 80.95% at 325. The cross-domain comparisons use different internal policies, including trace-only on MBPP, static-minimal on MATH, and Fixed-Deep on QA. Their scores do not establish the complete repair method's benefit across domains. With so many variants and tables, it takes unnecessary effort to tell which gains belong to the full method.
The writing in this paper omits context needed for human readers, while creating unnecessarily long, repetitive basic claims and presenting long lists of experimental numbers with little interpretation.
Questions for authors
-
How should a user choose between the full method and closed-draft replay, given the equal recovery count and the full method's lower false-repair rate but higher cost? Which naturally occurring failures actually require probing?
-
What explains the gap between the controlled-fault results and the lack of improvement on natural upstream failures? Are the failure types different, or do the adapters lack the necessary interventions?
-
How does the complete repair method compare with Trace-Repair on mathematics and QA, using the same starting workflow and matched resource budgets? Table 7 uses different internal policies across datasets.
-
For a new workflow, which interventions and replay plans must be specified by hand? How much of the existing adapters can be reused?