acceptodds
Withdrawn from ICLR 2027

Beyond Crowd Labels: Learning Process Supervision from Crowd Reasoning Trajectories

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

22 views

Abstract

Collecting high-quality supervision data is critical to the development of modern AI systems, and crowdsourcing provides a scalable way to obtain such data from multiple workers. Classical learning-from-crowds methods recover reliable labels from noisy worker annotations over predefined response options. Recent advances in large language models have increased demand for supervision that captures not only what the correct answer is but also how it is derived, which motivates the crowdsourced collection of free-form reasoning trajectories. In this paper, we ask whether, when crowd workers provide free-form reasoning trajectories instead of annotations over predefined options, reliable process supervision can still be recovered directly from noisy and heterogeneous worker contributions. To answer this question, we propose a structured inference framework that leverages shared reasoning structure across heterogeneous worker trajectories. We first construct a task-specific latent consensus process and softly align each worker trajectory to this process to establish cross-trajectory correspondences. We then perform structured variational inference over the aligned trajectories and observed final outcomes to recover the validity of individual reasoning steps. Evaluations with both LLM and human crowdworkers show that reliable step-level validity signals can be recovered from heterogeneous crowd reasoning trajectories and verified final outcomes without external step-level evaluators. The recovered signals capture meaningful variation in intermediate reasoning validity and provide useful supervision for downstream LLM fine-tuning across multiple reasoning domains.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
human-ai interaction, human-centered evaluation, human data collection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pTfuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies whether step-level process supervision can be recovered from heterogeneous free-form reasoning trajectories when only final answers are verified. It first aligns trajectories to a task-specific latent consensus structure using fused unbalanced Gromov–Wasserstein transport, then applies structured variational inference with worker-specific transition and outcome parameters to estimate step validity. Experiments cover five reasoning benchmarks, downstream fine-tuning of two language models, and a smaller human-crowd study.

Critical strengths
  • Important problem and thoughtful formulation. Recovering process supervision without annotating every reasoning step is valuable, and the proposed alignment framework accommodates variable-length trajectories, different solution paths, and error correction.

  • Broad empirical scope. The paper evaluates five reasoning domains, two downstream model backbones, several ablations, and human as well as LLM crowdworkers. This is considerably more comprehensive than a single-benchmark evaluation.

Critical weaknesses
  • The evaluated method is not actually free of step-level supervision. Appendix D.2 states that expert step-validity labels from 10 validation questions per domain are used for hyperparameter selection. This is consequential because the method is materially sensitive to these choices: for example, Table 8 shows GSM8K AUPRC changing from 61.8 to 80.7 when KK changes from 4 to 12. Therefore, the experiments do not establish the paper’s central claim that process supervision can be recovered without external step-level annotations.

  • The latent state is not convincingly identified as logical validity. The only direct correctness signal for an entire latent sequence is a binary terminal outcome. The monotonicity constraints in Equations 6–7 orient the latent states but do not show that they correspond to logical correctness rather than merely an outcome-predictive hidden state. This concern is amplified by the conditional-independence assumption: multiple synthetic “workers” may be prompt variants of the same underlying LLM and therefore share systematic errors. No identifiability result or stress test with correlated, duplicated, or majority-wrong workers is provided. This undermines the main interpretation of the inferred probabilities.

Questions for authors
  • What completely label-free procedure would select K,α,ρ,λvK,\alpha,\rho,\lambda_v in a new domain? Please report results using a configuration selected without any target-domain step-validity labels.

  • Under what assumptions is logical step validity identifiable from terminal outcomes alone? Can the authors provide either a formal recovery argument or controlled experiments with known latent states, including correlated and majority-wrong worker populations?

Reviewer_hANROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates an interesting and potentially important problem: whether reliable process supervision can be recovered from multiple noisy, heterogeneous, free-form reasoning trajectories without requiring expert step-level annotations or an external process verifier.

Unlike conventional learning-from-crowds settings, where workers annotate predefined labels, the proposed setting involves reasoning trajectories that may differ in length, structure, decomposition, and intermediate conclusions. The authors propose a structured inference framework that first constructs a task-specific latent consensus process and aligns individual trajectories to it using fused unbalanced Gromov–Wasserstein (FUGW) transport. The aligned trajectories, together with verified final-answer outcomes, are then used in a structured variational inference framework to estimate the posterior validity of individual reasoning steps.

The paper evaluates the proposed framework using both LLM-generated and human-generated reasoning trajectories. The results demonstrate substantial improvements in step-validity recovery over several existing approaches. The recovered validity probabilities are further employed as soft token-level supervision weights for fine-tuning Llama-3.1-8B-Base and Mistral-7B-v0.1, yielding consistent improvements across five reasoning benchmarks.

Overall, I find the research question meaningful and the proposed formulation interesting. The empirical results on process-supervision recovery are encouraging. However, I have several substantive concerns regarding the assumptions underlying the recovery process, the necessity of the proposed framework relative to alternative supervision strategies, and the limited evidence that improved step-validity estimation translates into meaningful downstream capabilities.

Critical strengths

S1. Recovering process-level supervision from noisy and heterogeneous contributions is substantially more challenging than conventional label aggregation, especially when different workers follow different reasoning paths. The proposed setting is relevant to the growing demand for scalable reasoning supervision, particularly when reliable step-level annotations are difficult or expensive to obtain. I view this problem formulation as one of the paper's main strengths. S2. The combination of cross-trajectory soft alignment, latent consensus modeling, worker reliability estimation, and sequential validity inference provides a coherent approach to the problem. Although the underlying techniques are individually established, their integration into a framework for heterogeneous reasoning trajectories is technically meaningful. In particular, modeling the evolution of reasoning validity is more expressive than assigning static reliability scores to workers or trajectories.

Critical weaknesses

W1. My primary concern is about the fundamental assumptions underlying the proposed framework.The method relies on structural correspondences and shared evidence across multiple reasoning trajectories. However, agreement among workers does not necessarily imply reasoning correctness, especially when workers share systematic misconceptions, common training data, similar prompting strategies, or correlated failure modes. This issue is particularly relevant to the paper's LLM-worker setting, where multiple workers may originate from the same underlying model under different prompt configurations. Such workers are not necessarily statistically independent. The paper demonstrates empirical agreement between inferred validity and human annotations on the evaluated datasets, but this does not establish when the latent validity states are identifiable from noisy trajectories and final outcomes.

W2. I am positive about the problem of recovering process supervision without expert step-level labels. However, I am less convinced that the paper sufficiently establishes when the proposed approach is preferable to alternative ways of obtaining supervision. In the main experimental setting, reasoning trajectories are generated by multiple LLM workers. This raises an important practical question: given access to multiple LLMs, why should one invest in collecting and structurally aggregating multiple noisy trajectories rather than using the same resources to obtain fewer higher-quality trajectories, employ a stronger model as a judge, or generate additional verification evidence?

W3.The paper reports substantial improvements in step-validity recovery, but the corresponding downstream gains over the strongest competing fine-tuning strategies are considerably smaller. For Llama-3.1-8B-Base, the proposed method improves MATH accuracy from 16.1% to 16.7% over the strongest baseline, and GSM8K accuracy from 47.3% to 47.8%. These correspond to improvements of only 0.6 and 0.5 percentage points, respectively. The improvements on LogiQA, P-FOLIO, and ZebraLogic are more noticeable, and the consistent positive results across both backbone models are encouraging. Nevertheless, the relatively small gains on mathematical reasoning raise an important question about the relationship between accurate process-supervision recovery and actual reasoning improvement.

Questions for authors

Q1. What assumptions are required for recovering reliable step validity from heterogeneous trajectories? In particular, how does the method behave when most workers share the same incorrect intermediate reasoning, or when multiple LLM workers exhibit strongly correlated errors? Are there identifiable conditions under which the recovered validity probabilities can be considered reliable?

Q2. How should we understand the gap between strong step-validity recovery and modest downstream improvements? For example, the improvements over the strongest fine-tuning baseline are only 0.6 percentage points on MATH and 0.5 percentage points on GSM8K for Llama-3.1-8B-Base. Do the authors have evidence indicating whether this gap is caused by limitations of the recovered supervision, weighted SFT, or the backbone model?

Q3. Does the method remain useful for stronger reasoning models? Can the authors evaluate the proposed supervision strategy using a stronger instruction-tuned or reasoning-oriented backbone, rather than only general-purpose base models? This would help establish whether the method offers complementary benefits beyond conventional reasoning training.

Reviewer_SwmuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses how to obtain step-level reasoning supervision from imperfect solutions when only final-answer correctness is available. It aligns related reasoning steps across multiple workers’ solutions, then uses a probabilistic model to estimate the probability that the reasoning remains valid after each step, combining cross-worker evidence, final-answer correctness, and learned worker reliability. During supervised fine-tuning, these probabilities weight the training loss on each step’s tokens, allowing useful intermediate reasoning from unsuccessful solutions to contribute while reducing the influence of likely errors. Across MATH, GSM8K, LogiQA, P-FOLIO, and ZebraLogic, the method improves step-validity prediction against human annotations and increases final-answer accuracy after fine-tuning Llama-3.1-8B and Mistral-7B. A separate human-crowd study evaluates step-validity prediction on human-written solutions. Its main contribution is a framework for recovering useful process supervision from heterogeneous crowd solutions without expert step labels or an external step-level judge.

Critical strengths
  1. The paper’s method combines evidence across multiple workers’ solutions to identify valid reasoning steps. Previous work already studies how to obtain step-level supervision from final-answer correctness, and this paper’s method can make use of both correct and incorrect traces. It matches related reasoning steps across already collected solutions, even when they appear at different positions or levels of detail. It then combines those matches with each solution’s final-answer correctness and learned estimates of worker reliability to infer where reasoning remains valid, goes wrong, or recovers. This method has novelty compared to prior works.
  2. The authors demonstrate that the inferred step-validity scores provide useful training supervision: using them to weight SFT improves final-answer accuracy across five benchmarks and two model backbones, compared with training on all solutions, filtering by final-answer correctness, and existing data-selection methods.
Critical weaknesses
  1. The downstream SFT experiments do not compare against the process-supervision baselines evaluated for step-validity prediction, such as Math-Shepherd, AutoPSV, PAV, and FreePRM. Consequently, it remains unclear whether the proposed method’s better step-validity predictions translate into greater final-answer accuracy gains than these alternatives. Using each method’s step scores to weight training under the same SFT setup would provide a more direct assessment of the proposed supervision’s practical advantage.
  2. Agreement across solutions may reflect a shared mistake rather than correct reasoning. Multiple workers, especially those based on the same model with different prompts, may repeat the same incorrect reasoning, potentially causing the method to assign high validity scores to those steps. Although final-answer correctness provides additional evidence, it does not identify which intermediate steps are wrong. The paper does not systematically test robustness to shared errors, such as whether the method can recognize correct reasoning when most workers repeat the same mistake.
  3. The experiments do not fully isolate the value of step-level weighting as opposed to other unit. A useful downstream control would weight every token in a trajectory by its average inferred validity, or by inferred worker reliability. Comparing these controls with per-step weighting would clarify whether locating errors within solutions matters beyond identifying generally better solutions or workers.
Questions for authors

see weakness 1. It would be helpful if the authors could provide these additional baselines for the SFT experiments section. See weakness 2: can authors provide an analyses of how often that common mistakes happen and whether that affects the method? see weakness 3. It would be helpful if the authors could provide additional control experiment on this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.