- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces TRUSTTRACE, which organizes programs into “task strategy–module descriptions–code implementations,” weights training examples by response-quality scores, and uses module execution traces to guide revisions. On a held-out test set from the authors’ benchmark, it outperforms standard modular SFT.
Critical strengths
- This paper proposes a framework called TRUSTTRACE for synthesizing data and optimizing model training.
- The method shows a modest performance improvement over standard SFT.
Critical weaknesses
- The paper applies some quality controls to the synthetic data, but does not establish that the generated code is reliably correct across all valid inputs.
- The fine-tuning experiments focus on Qwen2.5-Coder-7B and DeepSeek-Coder-6.7B, leaving TRUSTTRACE’s effectiveness on a broader range of open- and closed-source models unclear.
- The performance gains over standard SFT are modest. For example, Plan StrictAcc improves by only 1.00 percentage point, and it remains unclear whether this gain would persist at larger model scales.
Questions for authors
- Will the code and data for the self-constructed dataset be released?
- Why are comparisons with commonly used models missing for the self-constructed dataset?