- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ReComp, a framework for repairing failed coding-agent trajectories to generate additional training data at a lower cost. ReComp first identifies anomalous actions in failed trajectories, retrieves compatible action spans from successful trajectories, and recomposes them through a learned planner. The repaired trajectories are then executed and verified to ensure correctness. Experimental results show that ReComp can successfully repair a portion of failed trajectories, reduce training action NLL, and lower data acquisition costs compared to generating complete new trajectories.
Critical strengths
1. Novel and cost-efficient trajectory augmentation approach.
The paper proposes an interesting approach to reusing failed coding-agent trajectories by replacing anomalous actions with compatible actions from successful trajectories, reducing the cost of collecting additional training data.
2. Comprehensive experimental evaluation.
The paper evaluates anomaly detection, trajectory repair, downstream training utility, and data acquisition costs, providing empirical evidence for the effectiveness of the proposed framework.
Critical weaknesses
1. Limited improvement over Retrieval Top-1.
ReComp only improves the strict acceptance rate from 17.18% to 18.23% compared to Retrieval Top-1. This relatively small improvement raises concerns about whether the additional complexity of the planner is justified.
2. Lack of strong downstream training baselines.
The downstream training experiments lack comparisons with strong data selection and augmentation baselines, making it difficult to demonstrate the effectiveness of ReComp-generated training data over existing approaches.
Questions for authors
Mentioned in Critical Weaknesses.