- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Authors introduce a new benchmark called CDR-Bench that has 3,462 data-refinement tasks built from 29 Data-Juicer operators. A model gets a raw text, and a NL recipe of mappers/filters, and has to return the refined text with keep/drop decision. References came from replaying the operators. Covers single operators, unordered compositions, pair order swaps, and a filter placed before/between/after mappers. Evaluating across 15 LLMs, they report the recipe success falls on compositional and order sensitive settings. They use oracle diagnostics to locate filter errors. Finally, authors propose a state-aware prompting and a fine-tuned model called Juicer that improves the order swaps success from 7% to 32%.
Critical strengths
Paired design is a good idea. Order-M keeps only the swaps where reference actually changed, and Order-F holds the input/recipe fixed while moving the filter. Refs are deterministic and needed no LLM judge.
Critical weaknesses
- The results do not clearly separate procedural reasoning failures from text reproduction errors.
- Low mapper accuracy and whole-output scoring leave the source of the compositional drop unclear. Please compare matched component tasks and separate formatting deviations from incorrect transformations.
- On the paired diagnostic subset, filter-decision accuracy improves from Pre to Post for both models (Tab. 17). Lower complete-recipe success therefore does not by itself establish that later filtering decisions are harder. Please separate the contribution of incorrect KEEP/DROP decisions from errors in producing the required text.
- Proposed mitigations are in-domain solutions and overlaps with prior work.
- Juicer’s gains are explicitly dev-set results informed by benchmark feedback. They establish adaptation, but not generalization to unseen recipes or operators.
- The prompting strategies closely overlap with scratchpad/state-tracing and plan-and-solve approaches, while code-run uses established program-assisted execution. These connections need clearer citation... The compositional and multi-step execution breaks down is well established in Dziri et al., Faith and Fate, NeurIPS 2023; La Malfa et al., Code Simulation Challenges for LLMs, 2024. Code Simulation also proposes a line-by-line state-tracing prompt that's close to the state-aware prompting.
Then what's left is the paired order and placement design. That's not enogh when the metric doesn't isolate what the paper claims to measure.
Questions for authors
-
What portion of single-operator mapper failures is due to unstated conventions (e.g., formatting, deleting/placeholding) rather than faulty execution? Were fidelity of instruction and reachability of reference checked by humans?
-
What's the keep/drop counts per placement in Order-F? Why is Pre-status accuracy under 50% in Table 17, and why is Pre not in Fig 3?
-
Please report juicer performance on unseen recipes, operators, domains, and the full training details.