acceptodds
Withdrawn from ICLR 2027

Procedural Faithfulness in Data Refinement: Can LLMs Execute Compositional, Order-Sensitive Recipes?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Executing instructions that combine multiple operations requires following not only what to apply but also when. We study this procedural faithfulness in data refinement, where operation order and intermediate text states determine the correct output. Existing evaluations focus on individual tasks or end-to-end agent workflows, leaving the model's own ability to execute compositional, order-sensitive recipes insufficiently characterized. We introduce CDR-Bench, a diagnostic benchmark with 3,462 deterministic tasks across four domains and 29 operators, featuring paired execution-order and filter-placement variants. Across more than ten general-purpose LLMs, we find substantial gaps between individual-operation competence and complete recipe success in both rule-based and semantic refinement. Oracle interventions reveal difficulties in intermediate-text construction and filter-statistic computation, while code-assisted execution improves performance but leaves failures in recipe interpretation and order preservation. We develop training-free state-aware prompting that elicits explicit intermediate-state tracking and introduce Juicer, a specialized refinement model trained through broad operator learning followed by recipe-oriented adaptation. State-aware prompting improves compositional recipe success, while Juicer raises Order-M RS@3 from 7.02% to 31.82%. Overall, our study identifies procedural faithfulness as a distinct evaluation target in data refinement and provides a controlled framework for examining how models handle execution order, intermediate states, and stopping conditions.

Primary area:
datasets and benchmarks
Keywords:
procedural faithfulness, compositional instruction following, order-sensitive execution, data refinement, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6K35OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Authors introduce a new benchmark called CDR-Bench that has 3,462 data-refinement tasks built from 29 Data-Juicer operators. A model gets a raw text, and a NL recipe of mappers/filters, and has to return the refined text with keep/drop decision. References came from replaying the operators. Covers single operators, unordered compositions, pair order swaps, and a filter placed before/between/after mappers. Evaluating across 15 LLMs, they report the recipe success falls on compositional and order sensitive settings. They use oracle diagnostics to locate filter errors. Finally, authors propose a state-aware prompting and a fine-tuned model called Juicer that improves the order swaps success from 7% to 32%.

Critical strengths

Paired design is a good idea. Order-M keeps only the swaps where reference actually changed, and Order-F holds the input/recipe fixed while moving the filter. Refs are deterministic and needed no LLM judge.

Critical weaknesses
  1. The results do not clearly separate procedural reasoning failures from text reproduction errors.
  • Low mapper accuracy and whole-output scoring leave the source of the compositional drop unclear. Please compare matched component tasks and separate formatting deviations from incorrect transformations.
  • On the paired diagnostic subset, filter-decision accuracy improves from Pre to Post for both models (Tab. 17). Lower complete-recipe success therefore does not by itself establish that later filtering decisions are harder. Please separate the contribution of incorrect KEEP/DROP decisions from errors in producing the required text.
  1. Proposed mitigations are in-domain solutions and overlaps with prior work.
  • Juicer’s gains are explicitly dev-set results informed by benchmark feedback. They establish adaptation, but not generalization to unseen recipes or operators.
  • The prompting strategies closely overlap with scratchpad/state-tracing and plan-and-solve approaches, while code-run uses established program-assisted execution. These connections need clearer citation... The compositional and multi-step execution breaks down is well established in Dziri et al., Faith and Fate, NeurIPS 2023; La Malfa et al., Code Simulation Challenges for LLMs, 2024. Code Simulation also proposes a line-by-line state-tracing prompt that's close to the state-aware prompting.

Then what's left is the paired order and placement design. That's not enogh when the metric doesn't isolate what the paper claims to measure.

Questions for authors
  1. What portion of single-operator mapper failures is due to unstated conventions (e.g., formatting, deleting/placeholding) rather than faulty execution? Were fidelity of instruction and reachability of reference checked by humans?

  2. What's the keep/drop counts per placement in Order-F? Why is Pre-status accuracy under 50% in Table 17, and why is Pre not in Fig 3?

  3. Please report juicer performance on unseen recipes, operators, domains, and the full training details.

Reviewer_7nKqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CDR-Bench, a diagnostic benchmark for evaluating how LLMs handle compositional, order-sensitive text refinement recipes. Through extensive evaluations of more than ten state-of-the-art LLMs, the authors demonstrate that high competence in individual, isolated operators does not translate to success in multi-step recipes, especially when operation order or deferred filtering is involved. They analyze failure modes, test prompting baselines, and present Juicer, a specialized model trained via a two-phase curriculum.

Critical strengths

Benchmark contribution: The benchmark covers 29 operators, four distinct domains, 3,462 tasks, and incorporates both rule-based and semantic extensions, representing a massive empirical effort.

Critical weaknesses

Novelty of the Core Finding: That LLMs struggle with multi-step compositional planning, order preservation, and state tracking is a known limitation in the broader agentic and reasoning literature. While CDR-Bench formalizes this specifically in the context of data refinement, the core qualitative insight—that models fail at compositional long-horizon state tracking—feels incremental relative to existing work on procedural reasoning (e.g., ProcBench, SIFo).

Sensitivity to Prompt Formatting: The paper relies heavily on 11 prompt styles to compute RS@3. However, data-refinement tasks in production are rarely presented in 11 stylistic variations; they are typically structured specifications. The variance across styles suggests that model success is highly brittle to minor formatting choices, which points as much to prompt sensitivity issues as it does to an inherent lack of procedural faithfulness.

Novelty in Juicer: Juicer is essentially a standard multi-stage SFT + RL pipeline applied to Qwen3.6-35B-A3B using Data-Juicer data. Given that the benchmark operators are directly pulled from Data-Juicer, it is somewhat unsurprising that training on Data-Juicer style examples improves performance on a benchmark derived from Data-Juicer.

Questions for authors

model scaling should be studied (e.g., dense model size, MoE architectures, etc.) and the specific failure of ignoring execution order (Table 4)

Reviewer_AmU7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CDR-Bench, a benchmark of 3,462 deterministic data-refinement tasks spanning four domains and 29 Data-Juicer operators. Tasks are organized into single-operator tracks, an order-agnostic composition track, and two order-sensitive tracks: one swaps mapper order, the other moves a filter to different positions in the recipe. Given an input text and a natural-language recipe, the model must output the refined text and a KEEP/DROP decision, which is scored against executable references by exact-match Recipe Success and an edit-distance-based Refinement Gain. Experiments on about fifteen LLMs with reasoning disabled show that success drops from single operators to composed recipes and drops further when order matters. Oracle interventions attribute most filtering errors to computing text statistics. Code execution and state-aware prompting help only partially. A fine-tuned model, Juicer, raises Order-M RS@3 from 7.02% to 31.82%.

Critical strengths
  • The design is controlled and diagnosable. Paired order variants share the same input and operator set, and executable references give deterministic intermediate states. Together these let the authors attribute failures to specific causes, which end-to-end agent benchmarks cannot do. The Gold State / Gold Statistic oracle cleanly separates “constructing the intermediate text” from “computing the statistic” from “applying the rule.” This is the paper’s most convincing contribution.
  • The evaluation is broad and careful. The study covers many recent models with bootstrap CIs, 11 instruction styles, length analysis, and code-run and prompting baselines. Limitations, such as treating Juicer results as in-domain evidence, are stated honestly.
Critical weaknesses
  • The benchmark may measure spec-matching and counting more than procedural faithfulness. Even single-operator tasks peak at about 30% RS@3. This suggests the natural-language instructions underspecify exact Data-Juicer behavior, such as whether to delete or use placeholders, and how to handle leftover whitespace. Exact-match scoring then penalizes reasonable interpretations. For Order-F, the oracle shows near-perfect accuracy once the statistic is given, so much of the deferred-filter difficulty may come from LLMs’ known weakness at precise character or word counting rather than from ignoring execution order. Atomic and compositional tasks also use different inputs, so the composition gap is not cleanly identified.
  • The evaluation setting limits the conclusions. Reasoning is disabled in all main results, yet thinking mode raises Order-F by about 18–20 pp, so the headline claims hold only in a restricted regime. Juicer is trained with benchmark feedback and evaluated without held-out recipes or operators. Its gains therefore mainly reflect fitting to Data-Juicer conventions.
Questions for authors
  • Can you provide a matched-input analysis, per-step success on the same inputs used in compositional recipes, compared with the independence prediction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.