acceptodds
Under review as a conference paper at ICLR 2027

OFFIDEX: Training Office Assistants to Clarify Before They Deliver

Abstract

The professionalism of a senior office worker rests on two abilities: producing deliverables that satisfy acceptance criteria, and eliciting, before any work starts, the constraints that decide whether a deliverable will be accepted. Large language model (LLM) assistants excel at the former on well-specified prompts but habitually answer first on the underspecified requests that dominate real workplaces, silently guessing deadlines, audiences, formats and data sources. We present OFFIDEX, an office assistant agent trained to clarify before delivering, and a Recursive Self-Purifying Experiential Reinforcement Learning framework that turns clarification into an optimizable objective without a single ground-truth answer. Three tiers of models cooperate: a frontier curator that purifies raw community knowledge into 125,982 task cards with hidden constraints and acceptance rubrics, mid-tier executors that act as a reveal-only-when-asked user simulator and a two-tier rubric/process judge, and a lightweight learner that improves through supervised fine-tuning and reward-filtered self-training. Every stage is guarded by a purification gate (data leakage, simulator fidelity, judge reliability, and a best-of-n gate that measures the learnable group-relative signal before any policy update). After purified fine-tuning the calibrated composite reward rises from −0.224 to +0.012 on validation prompts and stays positive (+0.020) on a fully held-out OfficeClarify, zero-clarification dialogues fall from 24% to 3–10%, and protocol violations fall from 2.06 per dialogue to zero. Five frontier models given the same clarify-then-deliver prompt, simulator and judge all score significantly below the trained assistant, answering first on 12–86% of requests: the habit is trained, not prompted. The loop’s diagnostics further show that hard hierarchical vetoes collapse open-ended office rewards onto a few fixed values and must be softened, and that reward is an inverted-U function of the number of clarifying questions, with one to two targeted questions optimal.

Primary area:
reinforcement learning
Keywords:
LLM agents, agentic reinforcement learning, clarifying questions, reward specification, process rewards

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kSWxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents OFFIDEX, a model trained to ask clarifying questions as an office assistant before responding. The proposed approach to train the model is based on recursive self-improving RL recipe which involves a hierarchy of models.

Critical strengths

None

Critical weaknesses
  • The paper is very poorly written and lacks clear motivation. As an example, the related work section is overly short and doesn't really outline how this work is different than the past cited work, or what closely related work exists in the literature.

  • The discussion of the results is not insightful; it mostly restates the numbers from the plots but does not explain trends or provide any new insights. The experiments also lack meaningful comparisons to previous baselines and standard approaches using LLMs.

Questions for authors

No specific questions.

1 comment
AuthorsOpenReview ↗

Related work, mechanism, and baselines

We thank the reviewer for the direct assessment. W1: "Writing, motivation, related work." We agree §2 is too compressed. The revision will expand it to about one page with four paragraphs (clarification/proactive dialogue; multi-turn agentic RL and clinical inquiry; rubric rewards, judges and reward hacking; office benchmarks and self-improvement), each ending with an explicit “difference from OFFIDEX” sentence. The introduction will lead with the two motivating numbers: 83% of 16,608 community questions are ≤20 characters; five prompted frontier models answer first on 12–86% of held-out requests. We will add a notation table and a glossary and remove the P1/P2, V1/V2, bo1/bo8 labels. W2: "Discussion and baselines." §6 will be rewritten around mechanisms: why a clinical hard veto collapses a graded office reward; why the clarification budget is inverted-U; why positive-only RFT fails. For baselines, the main text will include a compact Table A25: five frontier models under the same prompt, simulator and judge score 0.12–1.21 lower on paired differences, with 12–86% zero-Q and 0.7–6.8 protocol violations, while passing more rubric items when they deliver. We will present base/SFT/RFT and the reward ablations in one table. ReAct/Reflexion are not direct baselines because they lack a clarification action and a hidden-information simulator; we will cite them as context. Stage IV remains unvalidated; the contribution is the environment plus the diagnostic findings.

Reviewer_e6ZjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper claims that LLM assistants excel at producing acceptible deliverables on well-specified prompts, but tend to answer without clarification on the underspecified requests which dominate real workplaces. Targeting on this issue, it proposes OFFIDEX, an office assistant agent trained to clarify before delivering, together with a recursive self-purifying experiential RL framework.

The proposed OFFIDEX agent has a three-tier hierarchy: the curator, the executors, and the learner. The curator is a frontier model that converts raw knowledge-base articles and community questions into task cards and dialogues. The executors are mid-tier models serving as user simulator, grader, and auditor. The learner is a lightweight model to be trained. The proposed recursive self-purifying loop contains six stages: purify, fine-tune, roll out, grade, select / optimize, and evaluation / diagnose. The reward is based on two tiers: outcome tier measured by rubric pass ratio, and process tier broken down into eight dimensions.

The results section is hard to read. Results seem to demonstrate that SFT improves the overall reward, but the main table (table 1) is hard to interpret, without a clear baseline setup and apple-to-apple comparison. The appendix tables are frequently referred to in the main body, which may contain useful information, but the lengthy appendix without a clear structure makes it hard to extract useful information from there. Further studies are conducted to analyze reward rules, model behaviors, user simulators and judges. However, it is unclear whether and how they impact model performance. Notably, seems that GRPO is configured but not executed, leaving a loose end to the paper's execution.

Critical strengths

None

Critical weaknesses

1. The claimed contributions are not fully supported.

OFFIDEX is claimed to be "the first office assistant for which requirement clarification is an explicitly rewarded multi-turn objective". However, that the requirement clarification is an explicit reward does not necessary mean that it is a better-performing office assitant, especially in terms of the final deliverable quality.

The paper also listed "A recursive self-purifying experiential RL framework" as a major contribution, but as far as I understand, this RL framework is not complete in this work. GRPO is discussed, configured but not run (according to Appendix B.5: "Its execution is deferred to the next loop iteration because the compute tier required for RL jobs on the platform was not provisioned within this iteration"). Therefore, it remains unclear how effective this RL framework would be.

The third contribution talks about several quantitative analysis, which may be beneficial, but it is unclear how these findings can direct model improvements.

2. The presentation makes it diffucult to assess the paper's contributions.

The paper is not rigorously written. In equation (4), while can be inferred, it lacks a definition for wiw_i and SiS_i. At the end of section 4.3, the paper refers to "G0" and "Stage III", which are defined after these references (in section 4.4 and section 4.5).

While these are not major issues, the result section (section 6) is even harder to read. Table 1 mixes some components from V2 sampling with V3 results, making the comparison difficult to interpret. The writing of section 6.2 through section 6.5 are hard to follow. Statement like "Behaviour moved; quality did not" in section 6.5 is not easily understandable as a takeaway sentence. Appendix tables are frequently referred to (for example, section 6.3 refers to Table A11 and A12), indicating that the authors need to change the organization and make sure that the essentially important results are in the main body. The frequent use of self-defined jargons (e.g. P1/P2, V1/V2, G1/G2, bo1/bo8, Stage I/II) also adds to the reading difficulty, requiring readers to cross-reference all the time. Besides, figures 3 - 5 are not clearly annotated and are hard to interpret as well.

Questions for authors
  1. Does SFT improve final deliverable quality? Table 1 reports some SFT components from V2 sampling alongside V3 results. Is there a directly comparable outcome-reward result for the base and SFT models?

  2. How is appropriate non-clarification evaluated? Any evidence that the model learns when to clarify, rather than simply becoming more likely to ask questions, given a clarify-first prompt?

2 comments
AuthorsOpenReview ↗

Direct answers, corrected comparison, and re-scoping

We thank the reviewer for the detailed audit. W1–Q1: "GRPO not run; does SFT improve quality?" You are right. Stage IV was configured but not executed (Appendix B.5); we claim no RL gain. On 49 paired prompts, base vs SFT outcome is 0.082 vs 0.049 (paired −0.033 [−0.056, −0.010]; first-sample n=46: −0.043 [−0.069, −0.017]); its gain is behavioural (zero-Q 24.0%→3.1%, violations 2.06→0). We will re-scope as a diagnostic study and add a validated/not-validated table. Q2: "Appropriate non-clarification?" On OfficeClarify, 7/68 (10.3%) deliver without asking; suggestively (n=7), outcome is higher (0.086 vs 0.036) but process lower (−0.220 vs +0.070). Question count is flat (1.86/1.85/1.80 on 0/1/3 hidden); 69.5% of complete-card trajectories ask >1 — it learned “ask”, not “when to stop”. Table A12: +0.041 vs −0.171 on 2–3 hidden; advantage peaks at 1–2 questions (−0.526/−0.002/+0.063/−0.008/−0.402). We will add a necessity dimension, a question-efficiency sentinel, and a larger zero-Q study. W2: "Findings → improvements." Calibrate reward shape (V1 62.8%→5.3% pinned); reward question relevance, not count; use a group-relative optimizer that learns from the 44.6% negative-advantage trajectories. Minor: "Eq. 4; Table 1; §6; jargon; figures." Define S_i/w_i/S_safety; move G0/Stage III before first use; replace Table 1 and explain the V2/V3 tier mixing; rewrite §6 with explicit takeaways; move A11/A12 into the main text; add a glossary; annotate Figures 3–5.

Thanks for the reply. The additional data confirms my concerns (SFT decreases outcome reward; the RL framework was not executed, and the model learns to "ask", not "when to stop"). The suggested changes are encouraged, but they will not substantially change the work. I maintain my scores.

Reviewer_qKy4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work focuses on the ability of office assistants to clarify requirements before proceeding with task execution. The authors propose OFFIDEX—an intelligent office assistant agent trained to clarify requirements prior to delivery—along with a benchmark dataset, OfficeClarify, designed to evaluate an assistant's requirement-clarification capabilities. The effectiveness of this approach is preliminarily validated on the benchmark using a method called "recursive self-purifying experiential RL." The primary contribution of this work lies in identifying a specific challenge within office environments and proposing a comprehensive framework to both address and measure it.

Critical strengths

This work focuses on how to improve the clarification ability of the model in office scenarios. This question is very practical and related to practical applications, and has strong practical value.

Critical weaknesses

Preliminary verification shows that SFT can improve the clarity ability of the model, but the effectiveness of the proposed RL scheme has not been fully demonstrated in experiments.

Questions for authors

The effectiveness and value of Recursive Self Exploring Experiential RL compared to SFT in this task can be elaborated in detail.

1 comment
AuthorsOpenReview ↗

Scope, evidence, and the unrun Stage IV

We thank the reviewer for the precise question about the RL stage. W1: "RL scheme not fully demonstrated." You are right. Stage IV (GRPO) was configured, unit-tested (41/41) and gated, but not executed; Appendix B.5 says so. We claim no RL gain. The validated chain is narrower: Stage I changes behaviour (zero-Q 24.0% → 3.1%; protocol violations 2.06 → 0), Stage II shows a group-relative signal (Δ8 = 0.352; 94% of groups contain a positive sample), and Stage III is a negative control (RFT −0.033, p = 0.708). We will re-scope title/abstract/contributions, add a validated/not-validated table, and restore Limitations. Q1: "Value of RL compared with SFT?" (1) SFT imitates curator dialogues: it teaches format and habit but not deliverable quality (same-prompt outcome 0.082 → 0.049; paired −0.033 [−0.056, −0.010]). (2) RFT is a stronger self-generated baseline: top-2 per group from 2,383 rollouts (534 dialogues; mean 0.300), yet flat on OfficeClarify (−0.033, p = 0.708) and only raises questions (1.72→1.94). (3) GRPO uses all eight samples, including the 44.6% with negative advantage; 99.0% of groups contain both signs; the top-2 RFT imitates average +1.14 and the discarded rest −0.38; mean advantage peaks at 1–2 questions (−0.526/−0.002/+0.063/−0.008/−0.402). This is a mechanism argument, not a measured gain. What we will add. A stage-by-stage comparison table (SFT/RFT/GRPO: learnability, cost, evidence) and the paired base-vs-SFT result in the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.