ReproSaddler: Reproducing Agent Failures for Harness Optimization

Failures of deployed language-model agents reveal concrete weaknesses in their prompts, tool interfaces, and execution logic, but they are difficult to reuse for harness optimization. The interaction is often observable while the persistent environment that conditioned the failure stays private. Copying that state is not permitted, and an error label alone discards the context that made the failure informative. We introduce ReproSaddler, a framework for converting such evidence into fresh, executable failure scenarios. A trusted compiler fixes a de-identified oracle–agent failure relation and a value-free workflow blueprint, which the generator cannot revise. The generator establishes a scaffold for the required workflow in a retrieved donor environment, then proposes individually switchable task and environment conditions. Strict oracle validation and an all-conditions-OFF success control distinguish an executable task from the conditions under which the seed harness reproduces the target relation. Trace-guided refinement and conservative ablation reduce those conditions, so an accepted scenario targets the specific observed discrepancy rather than an arbitrary error. We formulate a six-arm evaluation that compares generated scenarios with observable-only evidence, random tasks, unrelated failures, privileged original tasks, and the unmodified harness. The primary endpoint is repair of the original user failures, which makes failure reproduction a testable data-generation claim rather than a proxy for agent improvement.