acceptodds
Under review as a conference paper at ICLR 2027

Analyzing Preference Optimization with Synthetic Contrastive Reasoning Traces for Multi-Table Q&A

Abstract

Multi-table question answering requires models to retrieve relevant evidence, link schemas, and perform compositional reasoning across relational tables. Preference optimization over paired correct and incorrect reasoning traces is ideally suited for teaching LLMs such capabilities, but it is unclear which underlying mechanisms are most critical for this task: the training objective, or the properties of the contrastive reasoning traces. We study this question using synthetic contrastive reasoning traces for MMQA, consisting of validated positive traces and plausible negative traces generated by heterogeneous LLMs, and three open-weight LLMs (Qwen3-14B, Mistral-8B, and Llama-3.1-8B) evaluated on three table Q&A benchmarks, plus an out-of-distribution multi-table evaluation set built from BIRD. To investigate the role of optimization methods, we hold the preference pairs fixed and compare DPO, DPO with an NLL regularizer, and reference-free Contrastive Preference Optimization (CPO). We then further analyze the role of contrastive data diversity, quality, and source of the traces: whether heterogeneous generators for positive and negative traces strengthen the contrastive signal, how correct, faithful, and coherent the traces are under automated and human evaluation, and whether open-weight generators can replace proprietary ones. We find that CPO achieves absolute average improvements over Q&A supervised fine-tuning ranging from 10.7 to 19.3 percentage points, while DPO on the same pairs degrades two of the three models relative to positive-trace SFT and is only partially repaired by regularization. Removing the reference model improves every model over regularized DPO and is the larger of the two changes for all three; generating negatives with a different LLM than the positives gives a small but directionally consistent advantage; and an open-weight generator comes within 1–2 points of the proprietary one on every comparison. A case study and out-of-distribution transfer to three-table queries indicate that the negatives encode realistic multi-table failure modes, such as skipped constraint verification, that the trained models learn to reject.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multi-table question answering, Synthetic data generation, Contrastive reasoning traces

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9kyROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies preference optimization for question answering over multiple tables. It constructs synthetic pairs of positive and negative reasoning traces and analyzes the roles of the training objective and data generation. The authors compare DPO, DPO+NLL, and CPO with the same data and SFT initialization. Experiments on three models and several table benchmarks show that CPO improves over SFT on answers or positive traces, including on BIRD and questions involving three tables. Generator comparisons and human audits further assess the quality and usefulness of the synthetic data.

Critical strengths
  1. The motivation is clear. Reasoning over several tables can fail through missed constraints or incorrect joins, so studying supervision from plausible errors is relevant to the task. The paper separates questions about optimization from those about data quality. This structure makes the analysis easy to follow, and the authors acknowledge several limitations of their evidence.

  2. The experiments cover several models and benchmarks, including transfer beyond the training distribution. The objective comparisons use the same data and initialization, with DPO+NLL providing a useful intermediate control.

Critical weaknesses
  1. The paper suggests that removing the reference model is the main reason for CPO's better performance. However, CPO and DPO+NLL use different likelihood margins with the same β value. CPO also uses sequence likelihoods without length normalization, while negative traces are longer than positive traces. These factors make it difficult to identify the actual source of the improvement.

  2. The benefit of negative reasoning traces is not well established. CPO includes an NLL loss on positive traces, but the paper does not provide a positive-only training baseline with the same training budget. Also, the negative traces contain changes at every reasoning step. The model may learn to detect these differences rather than actual reasoning errors. The current experiments do not fully separate these effects.

Questions for authors
  1. The current results show that removing the reference model helps when NLL regularization is included, but it is unclear whether this also holds without NLL. How does reference-free DPO without NLL perform?
  2. The case study is helpful, but it is unclear whether CPO reduces these reasoning errors more broadly. Compared with Trace SFT, does CPO make fewer errors in joins, constraint checking, and aggregation? Can you provide an error breakdown or a few cases?
1 comment

Since the authors have not given any response to my comments, I will maintain my original score here.

Reviewer_MaBmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the importance of preference optimization objectives and contrastive reasoning trace properties for multi-table question answering. The authors construct synthetic positive and negative reasoning traces from LLMs and compare DPO, regularized DPO, and reference-free CPO across three open-weight models and multiple benchmarks. Results show that CPO consistently outperforms supervised fine-tuning and DPO, while using different generators for positive and negative traces provides modest additional gains. Further analyses examine trace quality, generator choice, and out-of-distribution generalization, suggesting that contrastive training helps models avoid common multi-table reasoning errors.

Critical strengths

Extensive experiments: The paper evaluates multiple LLMs across different benchmarks.

Important topic: The paper studies an important question about preference optimization for multi-table reasoning.

Critical weaknesses

Limited novelty: The paper primarily presents empirical comparisons and analyses of existing preference optimization methods, without introducing new methods or substantially new insights. Several related aspects have already been extensively studied, including preference optimization with reasoning traces (Wang et al., ACL 2024; Lahlou et al., NAACL 2025), its application to structured reasoning tasks (Liu et al., ACL 2025), and the effects of preference data quality and contrastiveness (Pan et al., NeurIPS 2025; Djuhera et al., ICLR 2026). The authors should better clarify how their findings differ from these prior studies and what novel insights they offer beyond the multi-table QA setting.

  1. Wang et al. (ACL 2024) — Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning.
  2. Lahlou et al. (NAACL 2025) — PORT: Preference Optimization on Reasoning Traces.
  3. Liu et al. (ACL 2025) — Uncovering the Impact of Chain-of-Thought Reasoning for Direct Preference Optimization: Lessons from Text-to-SQL.
  4. Pan et al. (NeurIPS 2025) — What Matters in Data for DPO?
  5. Djuhera et al. (ICLR 2026) — When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets.
Questions for authors

See weakness.

Reviewer_N3zHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies preference optimization for multi-table question answering using synthetic contrastive reasoning traces. The authors generate validated positive traces and plausible negative traces for MMQA, then compare Q&A SFT, Trace SFT, DPO, DPO+NLL, and CPO on Qwen3-14B, Mistral-8B, and Llama-3.1-8B. Evaluation is conducted on MMQA, MMTU, TableBench, and an out-of-distribution BIRD-based multi-table set, with additional analyses of trace diversity, trace quality, generator choice, and three-table transfer.

Critical strengths
  1. The paper studies a relevant and underexplored setting. Multi-table Q&A requires schema linking and cross-table reasoning, and contrastive reasoning traces are a reasonable supervision format for this problem.

  2. The comparison among preference objectives is useful. Holding the preference pairs fixed while comparing DPO, DPO+NLL, and CPO helps isolate how the optimization objective affects performance.

  3. The experiments cover multiple models and datasets. The paper evaluates three open-weight LLMs on several table-QA benchmarks and includes an OOD BIRD-based evaluation, which makes the empirical analysis more informative.

Critical weaknesses
  1. The methodological novelty is limited. The paper mainly applies existing preference-optimization objectives to synthetic reasoning traces, so the contribution is closer to an empirical study than a new method.

  2. The interpretation of CPO’s advantage is confounded by sequence length and objective design. Since negatives are longer than positives and CPO uses a raw likelihood difference over the trace, part of the margin may reflect length or formatting effects rather than the model learning to reject incorrect multi-table reasoning.

  3. The evidence that models learn specific multi-table failure modes is still limited. The case study is helpful, but the paper does not provide a systematic error-type evaluation showing that trained models specifically improve on skipped constraints, wrong joins, schema-linking errors, or aggregation mistakes.

  4. The comparison does not fully rule out simpler uses of the same synthetic traces. Beyond Trace SFT, stronger alternatives such as hard-negative supervised training, error-type-specific data augmentation, or filtering/reranking-based training could test whether preference optimization is necessary for the observed gains.

  5. The evidence for scaling from two-table training to more complex multi-table reasoning is not yet strong. The three-table BIRD subset contains only 16 examples, so the reported transfer results are too small to substantiate broad claims about generalization to harder relational settings.

Questions for authors

Please see Critical Weaknesses. I would be happy to reconsider my rating if the authors can provide convincing responses to these concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.