- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies preference optimization for question answering over multiple tables. It constructs synthetic pairs of positive and negative reasoning traces and analyzes the roles of the training objective and data generation. The authors compare DPO, DPO+NLL, and CPO with the same data and SFT initialization. Experiments on three models and several table benchmarks show that CPO improves over SFT on answers or positive traces, including on BIRD and questions involving three tables. Generator comparisons and human audits further assess the quality and usefulness of the synthetic data.
Critical strengths
-
The motivation is clear. Reasoning over several tables can fail through missed constraints or incorrect joins, so studying supervision from plausible errors is relevant to the task. The paper separates questions about optimization from those about data quality. This structure makes the analysis easy to follow, and the authors acknowledge several limitations of their evidence.
-
The experiments cover several models and benchmarks, including transfer beyond the training distribution. The objective comparisons use the same data and initialization, with DPO+NLL providing a useful intermediate control.
Critical weaknesses
-
The paper suggests that removing the reference model is the main reason for CPO's better performance. However, CPO and DPO+NLL use different likelihood margins with the same β value. CPO also uses sequence likelihoods without length normalization, while negative traces are longer than positive traces. These factors make it difficult to identify the actual source of the improvement.
-
The benefit of negative reasoning traces is not well established. CPO includes an NLL loss on positive traces, but the paper does not provide a positive-only training baseline with the same training budget. Also, the negative traces contain changes at every reasoning step. The model may learn to detect these differences rather than actual reasoning errors. The current experiments do not fully separate these effects.
Questions for authors
- The current results show that removing the reference model helps when NLL regularization is included, but it is unclear whether this also holds without NLL. How does reference-free DPO without NLL perform?
- The case study is helpful, but it is unclear whether CPO reduces these reasoning errors more broadly. Compared with Trace SFT, does CPO make fewer errors in joins, constraint checking, and aggregation? Can you provide an error breakdown or a few cases?
Since the authors have not given any response to my comments, I will maintain my original score here.