- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors presented Adaptive Structured Transformation (AStrucT) where they transform raw documents into domain specific structure document that follows a learned schema. Essentially they are rewriting document from the view point of an expert in a domain using an LLM. The authors argue that this can eliminate misaligned distribution in original documents with the target task, and that transformed documents are better source for finetuning. On BRIGHT dataset, they show that fine tuning on original documents reduced performance compared to no fine tuning whereas using transformed documents improved performance.
Critical strengths
- It consistently improved performance over baseline and direct finetuning. The gains are consistent across three model scales (0.6B, 4B, 8B)
- Mixing as well as prepending original + transformed document, both are worse than just using transformed data. This presents a stronger case for using transformed documents.
- The approach is simple and has no inference-time cost
Critical weaknesses
None to support rejection
Questions for authors
- How much of this is because we used a stronger model to rewrite?
- How can we confirm that the problem of HQ FT is that the synthetic data is misaligned?
Response to Reviewer Zipr
Thank you for the supportive review and for highlighting the mixing and prepending ablations.
Q1: Rewriter strength. We don’t believe rewriter strength is as important as how the LLM is instructed to rewrite. GPT-4.1 is used for all our experiments, both generic rewriting (only +0.19 over HQ) and AStruct (+3.96 over HQ).
Q2: Misalignment. Our empirical results establish negative transfer and suggest misalignment, and even our embedding analysis is consistent with the explanation that HQ data does not match the test distribution. To dig deeper though, we can also look at where the documents come from. ReasonIR-HQ uses BRIGHT documents as positives, generates queries from those documents, and then generates hard negatives conditioned on the query and positive document. Their generation prompt does not instruct the model to align the document to match style or distribution with the real BRIGHT queries or documents, however it does encourage matching domain. This further suggests that dataset misalignment is the main problem.