- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces a benchmark and a process reward model for evaluating candidate actions under different role-specific objectives. The model uses the active role’s objectives and constraints to score candidates and guide agent decisions.
Critical strengths
None
Critical weaknesses
1.The paper is difficult to follow, and the presentation needs substantial improvement in clarity.
2.The choice of Qwen2.5-1.5B as the backbone is not sufficiently convincing.
3.The paper contains numerous typo and formatting issues, such as “orderings Fu et al. (2026).”
4.The paper’s main contribution and novelty are unclear.
Questions for authors
See weaknesses.
3 comments
Reply to W2: Backbone Choice and Extended Evaluation
We chose Qwen2.5-1.5B to build a lightweight process evaluator that can be called repeatedly during agent decision-making. PolyPRM scores multiple candidates at each decision point, making both evaluation cost and selection performance relevant to its practical use. The 1.5B backbone with LoRA allows us to test whether effective role-conditioned process evaluation can be achieved at modest computational cost.
To further examine this choice, we extended the experiments to Qwen2.5-7B, Llama-1B, and Llama-3B. The models use the same data splits, training recipe, and seed settings, all with LoRA (r = 8). Results across eight task configurations are shown below; each task uses its corresponding metric, with higher values indicating better performance (↑).
| Task ↑ | Qwen2.5-1.5B | Qwen2.5-7B | Llama-1B | Llama-3B |
|---|---|---|---|---|
| Trading | 0.774 | 0.781 | 0.777 | 0.776 |
| Portfolio | 0.787 | 0.788 | 0.794 | 0.780 |
| Market Making | 0.839 | 0.831 | 0.825 | 0.834 |
| Job Negotiation | 0.719 | 0.728 | 0.719 | 0.723 |
| FinQA | 0.929 | 0.929 | 0.942 | 0.973 |
| Retail | 0.828 | 0.842 | 0.825 | 0.872 |
| Support | 0.719 | 0.714 | 0.819 | 0.811 |
| Negotiation | 0.813 | 0.831 | 0.813 | 0.841 |
The benefits of larger backbones are task-dependent, without uniform improvements across tasks. Qwen2.5-7B performs close to its 1.5B counterpart overall. Llama-3B improves on FinQA, Retail, and Negotiation, while performing worse on Portfolio. The benefits of mandate conditioning are retained across all four backbones, supporting its applicability across model sizes and families.
Computational cost also supports the smaller backbone as a default: training each task configuration on a single GPU takes approximately 8–17 hours for Qwen2.5-1.5B, compared with 12–47 hours for Qwen2.5-7B. Thus, Qwen2.5-1.5B offers a reasonable balance between performance and computational cost, and PolyPRM’s effectiveness is not confined to this backbone. We will include these extended results in the revised manuscript.
Reply to W4: Main Contributions and Novelty
The central idea is simple and practically meaningful: the same agent step in the same context can have very different value for different roles, so process evaluation must specify whose objectives it serves. A high score under a common task metric does not automatically imply that an agent’s choices satisfy a particular user’s objectives, constraints, and interests. Our paper addresses this gap.
The paper makes three contributions:
-
Problem formulation: role-conditioned process evaluation. We explicitly model candidate-process value as , where represents the role’s mandate. With the state, history, and candidate process held fixed, different mandates can imply different or even opposite rankings.
-
Benchmark construction: RIVA-Bench. We provide a benchmark covering 8 tasks, 50 roles, and 111,805 candidate targets. By holding decision contexts and candidate sets fixed while varying role objectives and corresponding utilities, it tests whether evaluators can identify role-dependent differences in value.
-
Method and empirical validation: PolyPRM. We use a simple shared evaluator to score candidate processes conditioned on the mandate, and validate this design through candidate selection, downstream tasks, and role ablations. For example, Table 3 shows that removing role names preserves role-dependent behavior, whereas removing the mandate yields zero MCA on both Retail and Email, supporting the importance of explicit objectives and constraints for role-conditioned evaluation.
As emphasized in the introduction, our contribution lies in formulating this evaluation problem, providing a controlled benchmark that makes role-dependent differences identifiable, and validating a simple, effective approach. The standard model architecture enables an economical implementation; the contribution encompasses the problem formulation, benchmark, and empirical findings.
Dear Reviewer 6z84,
Thank you for your review. We hope our clarifications of the paper’s organization and core contributions, together with the new backbone scaling experiments, address your concerns. If so, we would appreciate your consideration of a higher score. We would also welcome specific feedback on any passages or explanations that remain unclear and are happy to answer further questions.
Best regards,
The Authors
Reply to W1 & W3: Presentation Clarity and Formatting
We address these comments together because both concern the paper’s organization and presentation. The manuscript follows “problem formulation → benchmark construction → evaluation method → experimental validation.” It first explains how the value of the same candidate process varies with role objectives, then introduces RIVA-Bench to characterize these differences under fixed contexts and candidate sets. It subsequently presents mandate-conditioned PolyPRM and evaluates the design through candidate selection, downstream tasks, and ablations. In this organization, the mandate specifies the evaluation objectives, RIVA-Bench provides the benchmark, and PolyPRM implements the conditional scoring function.
We will correct citation-integration issues such as “orderings Fu et al. (2026)” and check spelling, citations, and formatting consistency throughout the manuscript.
The broader concern that the paper is difficult to follow does not yet identify specific sections or reasoning steps. Could the reviewer indicate which passages, concept definitions, or methodological explanations require clarification? Concrete examples would help us understand the concern precisely and make targeted improvements.