acceptodds
Under review as a conference paper at ICLR 2027

IR-Agent: Decoupled Alignment for Large Language Model Driven Earnings Call Responses

24 views

Abstract

Company-side earnings-call answering demands three concurrent desiderata: every cited figure must be grounded in the company's publicly disclosed materials, relevant available figures should be properly cited in replies, and responses should adopt managerial conversational register instead of plain document recitation. Satisfying all three simultaneously poses a substantial challenge for large language models (LLMs), as these objectives are inherently conflicting: models optimized to mimic human-like speech tend to invent unsupported numeric claims. In this paper, we first propose IR-Agent, an industry agent for high-fidelity, disclosure-consistent earnings-call response generation, whose architecture is the Decoupled Alignment Framework (DAF). The style module aligns the agent to authentic managerial utterances via supervised fine-tuning, with style signals deliberately excluded from reward computation to avoid reward hacking. The disclosure module regulates numeric citation behavior through advantage-weighted policy optimization, exclusively tuning numerical grounding preference while preserving learned linguistic style. The verification module leverages a rule-based numeric verifier to filter and rank sampled candidates, eliminating fabricated numbers; this same verifier also provides consistent reward signals during training. We construct IRCorpus, the first company-side earnings-call response corpus with verifier-supported numeric supervision, and devise a closed-form evaluation protocol that quantifies fabrication, omission and stylistic fidelity without relying on LLM-as-judge metrics. On 573 held-out questions from 20 companies unseen during fine-tuning, eight-candidate IR-Agent yields zero verifier-detected fabricated numbers with high usability of grounded figures, and matches human managerial register more closely than eight frontier baselines, which persistently suffer from fabrication or excessive omission.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, LLM Agents, Reinforcement Learning, Factual Grounding and Hallucination

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6m1TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

IR-Agent combines supervised fine-tuning, numerical reward training, and selection from eight candidate answers for company-side earnings-call responses. The paper also introduces IRCorpus and rule-based measures of numerical support, omission, and writing style.

Critical strengths
  1. The paper studies a clear practical task and builds a corpus with separate companies for training, validation, and testing.

  2. The three stages give a simple and understandable way to combine imitation, numerical training, and response selection.

Critical weaknesses
  1. The verifier ignores the financial item, period, sign, and unit, and even rejects the correct value 937 when the source uses 937.00 in Appendix A.8. Without an independent audit, its scores do not establish how often the financial claims are actually correct.

  2. Both SFT plus selection and full IR-Agent have zero detected fabrication on the test set. RLOO reduces test omission from 17.5% to 15.2% but raises validation omission from 16.9% to 20.8%, so its added value is not consistent across the two sets.

  3. The API baselines provide one answer per question, while IR-Agent selects from eight candidates using the evaluation verifier. The paper also cannot confirm that the API models received the same instruction, so this comparison does not establish an advantage under matched conditions.

  4. Every retained test reference contains numbers that pass the verifier, so the benchmark covers a selected group of questions. It does not establish how well the system handles missing information or decides when a reply without numbers is appropriate.

Questions for authors
  1. Is the restriction to previously disclosed numbers a company policy or a benchmark assumption?

  2. How do the strongest API baselines perform with the same prompt, eight candidates, and the same selection rule?

  3. How does the decoupled design compare with a method that uses both style and numerical rewards under the same training budget?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6m1T: requested API control and numerical sensitivity results

Dear Reviewer 6m1T,

Thank you for your constructive questions and for recognizing the clear task formulation, company-disjoint corpus, and three-stage design. Your suggestions helped us strengthen the experiments. We have prepared a revised manuscript and answer your three questions below.

Q1. Is the restriction to previously disclosed numbers a company policy or a benchmark assumption?

It is the evidence-boundary assumption of this benchmark and assistant: the system answers from the supplied company materials, and the verifier measures adherence to that boundary. We have clarified this in the prepared revision. It is not a universal company policy or a legal-compliance test.

The retained references contain at least one source-matching number, allowing us to study numerical participation under this evidence constraint. This defines the numerical-answer subset being evaluated; cases where materials lack an answer require a separate abstention evaluation.

Q2. How do strong APIs perform with the same prompt, eight candidates, and selection rule?

Following your recommendation, we completed this comparison on all 573 common questions with identical system instructions, materials, eight-candidate budgets, and stable selection:

Generator + N=8 selectorAnswers with unmatched numbersInformative-clean coverageNumerical omission
IR-Agent (4B)0/57384.82%15.18%
GPT endpoint1/57334.21%65.62%
Gemini endpoint17/57391.80%5.41%

Using only 4B parameters, IR-Agent achieves zero verifier-detected violations in the final selected answers while retaining 84.82% informative-clean coverage. Your suggested control therefore provides stronger evidence for its usefulness as a compact, numerically constrained system. Gemini provides higher coverage, while IR-Agent satisfies the numerical rule on every selected answer in this test; GPT has low violations but substantially lower numerical coverage.

Informative-clean coverage requires g>=1 and f=0 in the selected answer; omission means g=0. The prepared revision includes the complete N=1/2/4/8 results (Section 4.4 and Appendix B) and the shared instruction (Appendix A.4). These new API endpoint runs are separate from the historical single-response results. Candidate budgets and visible-output scoring length are controlled, while API sampling distributions and internal reasoning compute differ from local runs.

Q3. How does decoupling compare with joint style and numerical rewards under the same training budget?

Our design assigns register supervision to demonstrations and numerical behavior to subsequent adaptation. This motivation is supported by Ibrahim et al., Nature (2026), who find that warmth fine-tuning can reduce factual accuracy and increase sycophancy. Our application concerns managerial register and evidence-bound numerical use; separating responsibilities avoids directly treating surface style proxies as a complete communication-quality objective.

The present controls evaluate the contribution of numerical adaptation within this design. At N=1, mean unsupported numbers decrease from 0.353 to 0.206 (41.6%). At N=2, all-dirty pools fall from 45 to 20 (55.6%) and informative-clean coverage rises from 45.72% to 51.48%. These provide finite-budget evidence for the adaptation stage. A same-training-budget joint-reward arm remains open; the revision presents objective allocation as the evaluated design without claiming established superiority over joint rewards.

Your 937 versus 937.00 example

Thank you for highlighting this distinction. It motivated a sensitivity check that holds the N=8 selected answers fixed and converts extracted strings to Decimal values. Unsupported-number totals remain unchanged for Base, SFT, IR-Agent, and GPT; Gemini changes from 18 to 17. Thus, the main comparison is maintained after numerical-value normalization. We report this separately in Appendix B. The verifier establishes numerical membership, while independent semantic assessment must also check metric, entity, unit, sign, and period associations.

Adaptation across splits

Your observation helped us distinguish finite-budget effects from the final N=8 endpoint. Both SFT and IR-Agent reach zero detected violations at N=8; the revision retains validation and test results and emphasizes where adaptation improves candidate supply, rather than presenting omission improvements as uniform across splits.

Thank you again for helping improve the work. With approximately two days remaining in the discussion period, we would appreciate your feedback on these additions and your consideration of the new evidence when updating your assessment.

Best regards, The authors

Reference: Ibrahim, Hafner & Rocher. Nature (2026). https://doi.org/10.1038/s41586-026-10410-0

Thank you for the new experiments and for the revised tables. I have one question about the omission results.

Table 3 reports test omission of 17.5 percent for SFT plus selection and 15.2 percent for the full system. Table 6 reports 16.9 percent and 20.8 percent on validation. The direction is opposite in the two splits, and the two are reported apart.

If the two splits are pooled, SFT plus selection omits on 144 of 833 questions and the full system omits on 141 of 833. That is 17.3 percent against 16.9 percent, so the 2.3 point gain on test becomes 0.4 points once validation is counted.

Could you report this pooled comparison and say whether the difference remains meaningful? If you think the two splits should not be pooled, could you explain what makes them different enough to report separately?

AuthorsOpenReview ↗

Pooled omission comparison and clarification of the adaptation benefit

Dear Reviewer 6m1T,

Thank you for checking the two tables together and suggesting this useful summary. We have independently verified your calculation. At N=8, the split-specific and pooled omission counts are:

SplitSFT + selectionFull system
Test100/573 (17.45%)87/573 (15.18%)
Validation44/260 (16.92%)54/260 (20.77%)
Pooled144/833 (17.29%)141/833 (16.93%)

The pooled difference is 0.36 percentage points, or three questions. We agree that this small descriptive difference should not be interpreted as a meaningful overall omission advantage at N=8.

We originally reported the splits separately because validation was used for development and the companies are disjoint. This explains their different evaluation roles, but does not preclude a pooled descriptive summary. We will place the pooled comparison alongside both split-specific results, labeling it as a development-plus-test summary rather than an independent held-out estimate.

Your observation helps us clarify where disclosure shaping contributes most clearly: at limited candidate budgets. On the test set at N=2, under the same selector, it reduces all-dirty pools from 45 to 20 (55.6%) and increases informative-clean coverage from 45.72% to 51.48%. We will distinguish this finite-budget benefit from the similar aggregate omission of the two policies at N=8.

Thank you again for helping make the contribution and evaluation more precise.

Best regards, The authors

Reviewer_LSroOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces IR-Agent, a framework for generating company-side earnings-call responses that aims to balance numerical grounding, numerical participation, and managerial language style. Its Decoupled Alignment Framework combines supervised fine-tuning on filtered human responses, offline policy adaptation using RLOO-style advantages over a fixed pool of SFT-generated responses, and inference-time selection among eight candidates. A shared rule-based verifier checks whether numerical strings appear in the supplied materials, providing training rewards and selection criteria. The authors also construct IRCorpus with company-disjoint splits and evaluate numerical support, omission, and surface-level stylistic similarity. Experiments on 573 held-out questions compare IR-Agent with eight API-based models, while ablations examine training stages, candidate count, and reward design. The results demonstrate the importance of candidate selection in eliminating verifier-detected unsupported numbers, although this outcome reflects numerical-string membership rather than semantic factual correctness.

Critical strengths
  • S1: The paper establishes a useful empirical foundation for a practically important generation task. Company-side earnings-call responses must balance numerical grounding, informative disclosure, and managerial language style. IRCorpus pairs company materials with analyst questions and management responses, while company-disjoint splits support evaluation beyond the firms represented in fine-tuning. Together with separate measures of numerical support, omission, and surface-level style, this resource enables systematic investigation of trade-offs that a single factuality metric would miss. This task formulation and experimental foundation constitute a meaningful contribution even given the limitations of the proposed verifier.
Critical weaknesses
  • W1: The evaluation does not establish the semantic factual reliability central to the application. The verifier checks whether numerical strings occur in the supplied materials, without verifying their association with financial metrics, reporting periods, units, or entities. A response can therefore pass while attaching a disclosed number to an incorrect claim. Because the same verifier supplies training rewards, guides candidate selection, and measures numerical support, the reported zero-error result demonstrates conformity to the optimized proxy rather than independently validated factual correctness. The authors acknowledge this restriction, but without an independent semantic assessment, the evidence remains insufficient to establish reliability in financial communication.

  • W2: The baseline comparison does not establish the added value of specialized training. IR-Agent selects among eight sampled responses, whereas each external API baseline provides only one; identical system instructions also cannot be confirmed. Consequently, the comparison cannot distinguish the benefit of training IR-Agent from that of generating multiple candidates and applying the numerical verifier. Strong baselines evaluated with the same instructions, candidate budget, and selection rule are needed to determine whether the proposed training framework offers an advantage over existing models with equivalent post-processing. This missing control substantially weakens the empirical case for the method.

Questions for authors
  • Q1: The contribution of disclosure shaping remains insufficiently validated. This stage uses 8,000 SFT-generated responses with fixed advantages and no refreshed rollouts, making its learning signal dependent on the initial policy’s candidate coverage and the verifier’s limited reward. Semantically incorrect responses can receive positive advantages, while groups lacking correct responses provide no direct examples of the desired behavior. Although updates may generalize beyond the stored pool, the experiments do not establish that this stage consistently improves substantive answer quality. Its incremental results are also mixed: adding RLOO to SFT plus selection reduces test omission from 17.5% to 15.2%, but increases validation omission from 16.9% to 20.8%, while both configurations achieve zero verifier-detected unsupported numbers on the common test set. These findings leave the necessity and robustness of this central training stage unclear.
3 comments
AuthorsOpenReview ↗

Response to Reviewer LSro: controlled API comparisons and disclosure-shaping evidence

Dear Reviewer LSro,

Thank you for the detailed review and for recognizing IRCorpus and company-disjoint evaluation as a useful empirical foundation. Your comments helped us strengthen the controls and clarify the interpretation of the results. We have prepared a revised manuscript with the following additions.

W2. Same-instruction, same-candidate-budget comparisons

Following your suggestion, we generated eight candidates per question from two strong proprietary API endpoints on the same 573 questions, using identical system instructions, evidence, numerical verification, and stable selection:

Generator + N=8 selectorAnswers with unmatched numbersInformative-clean coverageNumerical omission
IR-Agent (4B)0/57384.82%15.18%
GPT endpoint1/57334.21%65.62%
Gemini endpoint17/57391.80%5.41%

IR-Agent uses only 4B parameters yet combines zero verifier-detected violations in its final selected answers with 84.82% informative-clean coverage. The control you proposed makes this operating point clearer: compared with GPT, IR-Agent supplies substantially greater numerical coverage; Gemini supplies higher coverage, while IR-Agent satisfies the numerical-membership rule on all selected answers in this test.

Informative-clean coverage requires g>=1 and f=0 in the selected answer; omission means g=0. The prepared revision reports N=1/2/4/8 for every generator (Section 4.4 and Appendix B). Selection benefits other generators too: coverage increases from 23.39% to 34.21% for GPT and from 80.28% to 91.80% for Gemini. Reporting these curves separates generator behavior from selection benefits.

The new API runs match prompts, evidence, candidate counts, visible-output scoring length, and the selector. API sampling defaults and internal reasoning compute differ from local runs, so we describe this as a matched-candidate comparison. Historical single-response API results are kept separate.

Q1. What does fixed-pool disclosure shaping improve?

The motivation for treating numerical behavior explicitly is supported by broader evidence that style adaptation and factual behavior can interact. Ibrahim et al., Nature (2026), find that warmth fine-tuning can reduce factual accuracy and increase sycophancy. Our task concerns managerial register and evidence-bound numerical use, motivating distinct responsibilities for register supervision and numerical adaptation.

To examine the adaptation effect you highlighted, we compare the policies under the same selector and candidate budgets. At N=1, adaptation reduces mean unsupported numbers from 0.353 to 0.206 (41.6%). At N=2, all-dirty pools decrease from 45 to 20 (55.6%), while informative-clean coverage increases from 45.72% to 51.48%. These results show a finite-budget candidate-supply effect alongside the final N=8 result, where both SFT and the complete system reach zero detected violations.

The update uses 8,000 fixed responses, reward r=g-2f, and leave-one-out advantages in signed advantage-weighted answer-token likelihood updates. Responses and advantages remain fixed; there is no learned value model. The revised method description states this precisely. Our task-specific contribution concerns the numerical reward and its interaction with finite-budget verification.

W1. Numerical membership versus independent semantic verification

Your distinction is helpful. We have made the verified property—source numerical-string membership—explicit in the prepared revision. Sharing this rule across training and selection specifies an operational target: if a pool contains a passing candidate, the selector delivers one. The experiments measure how often finite pools supply such answers while retaining numerical participation.

We consistently describe the result as zero verifier-detected unsupported numerical strings in the final selected answers. Independent claim-level checking of financial-item, entity, unit, sign, and period associations remains open. AI-assisted screening organizes evidence for this assessment; it is not a completed human evaluation. This distinction makes the established numerical result and the additional semantic assessment interpretable on their respective terms.

Thank you again for helping strengthen the work. With approximately two days remaining in the discussion period, we would appreciate your feedback on these additions and your consideration of the new evidence when updating your assessment.

Best regards, The authors

Reference: Ibrahim, Hafner & Rocher. “Training language models to be warm can reduce accuracy and increase sycophancy.” Nature (2026). https://doi.org/10.1038/s41586-026-10410-0

Reviewer_LSroOpenReview ↗

I will raise my score from 1 to 2.

Thank you for the detailed rebuttal and the additional experiments.

I acknowledge that the new matched-candidate comparisons largely address W2, substantially improving the fairness of the baseline evaluation. The additional candidate-budget analysis also provides meaningful evidence for the contribution of disclosure shaping under limited sampling budgets, partially addressing Q1.

However, W1 remains unresolved. While the revised manuscript more clearly distinguishes numerical-string membership from semantic factual correctness, this clarification does not substitute for independent claim-level verification. Consequently, the central concern regarding factual reliability in financial communication remains.

Furthermore, the incremental benefit of RLOO over SFT plus selection at N=8 remains modest, and the reported confidence interval includes zero, leaving the necessity and robustness of the full framework insufficiently established.

In recognition of the substantive experimental improvements, I will raise my rating from 1 to 2. Nevertheless, the unresolved semantic verification concern and the limited evidence for the necessity of the proposed alignment framework prevent me from supporting acceptance at this stage.

I appreciate the authors' efforts to strengthen the paper.

Thank you for taking the time to review the additional experiments. We appreciate your feedback on the matched-candidate comparison and the low-budget results, as well as your clarification of what remains open about claim-level factual accuracy.

Reviewer_hsyzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IR-Agent, a framework for generating earnings-call responses that balance numerical grounding, numerical participation, and managerial style. The framework combines supervised fine-tuning, RLOO-style advantage-weighted optimization, and verifier-guided Best-of-N selection. Experiments on a company-disjoint dataset report zero verifier-detected numerical fabrication after candidate selection.

Critical strengths

The paper addresses a practical problem in financial text generation, where factual consistency and communication quality are important. The company-disjoint dataset and separate evaluation of numerical fabrication, omission, and stylistic characteristics provide a useful application-oriented benchmark.

Critical weaknesses
  1. The paper primarily combines established techniques, including SFT, advantage-weighted optimization, and Best-of-N verification, without introducing a sufficiently novel method. Moreover, the main findings largely reflect expected properties of these techniques, such as reward-dependent trade-offs and candidate-pool coverage. It is unclear what generalizable research insights the paper contributes beyond this specific application.

  2. The numerical verifier only checks whether a number appears in the source materials, without validating its associated financial metric, time period, or semantic context. Consequently, the reported zero fabrication rate does not establish factual correctness. This is particularly concerning because the same verifier is used for training rewards, candidate selection, and evaluation.

  3. The independent contribution of the proposed alignment framework is not convincingly established. SFT with Best-of-8 selection already achieves zero verifier-detected fabrication, while the full framework only reduces omission from 17.5% to 15.2% on the test set, without a consistent improvement on validation. These results suggest that the main performance gains largely come from candidate selection rather than the proposed alignment procedure.

  4. The experimental comparisons are insufficient to establish the advantages of the proposed framework. IR-Agent uses eight sampled candidates with verification, whereas the API baselines generate only one response. Stronger comparisons under matched inference budgets are needed. In addition, direct comparisons against joint multi-objective optimization would be important to justify the proposed decoupled design.

  5. The evaluation of managerial style relies mainly on surface-level linguistic statistics, such as nominalization, deflection, and copying rates. These metrics do not adequately capture whether responses are natural, professionally appropriate, or semantically complete. The substantial length difference between generated and human responses further raises concerns about the validity of the style comparisons.

Questions for authors
  1. What is the main methodological contribution of decoupled alignment beyond combining existing techniques, especially given that SFT with Best-of-N already achieves the same zero verifier-detected fabrication rate?

  2. Can the authors provide semantic-level factuality evaluations and comparisons against joint multi-objective optimization under matched inference budgets?

1 comment
AuthorsOpenReview ↗

Response to Reviewer hsyz: new matched-candidate experiments and clarifications

Dear Reviewer hsyz,

Thank you for your thoughtful review and for recognizing the task's practical importance and the company-disjoint benchmark. Your questions helped us strengthen the experimental controls and clarify the contribution. We have prepared a revised manuscript incorporating the following evidence.

Q1. Main methodological contribution beyond existing techniques and SFT + Best-of-N

Our contribution is the task-specific allocation of register supervision, numerical adaptation, and output verification. Demonstrations teach how management responds; numerical rewards shape evidence-bound participation; the selector determines which candidate is delivered. “Decoupled” describes these responsibilities within a shared policy, not independent parameter subspaces.

This motivation connects to Ibrahim et al., Nature (2026), who find that warmth fine-tuning can reduce factual accuracy and increase sycophancy. In our setting, SFT moves several register proxies closer to human references while unsupported numerical strings increase from 0.127 to 0.353 per answer. This echoes the broader style–factuality concern and motivates explicit numerical adaptation, without attributing the change to a demonstrated sycophancy mechanism.

Your observation that SFT + Best-of-8 also reaches zero detection helped us focus on the incremental adaptation effect. With the same two-candidate budget, adaptation reduces all-dirty pools from 45 to 20 (55.6%) and increases informative-clean coverage from 45.72% to 51.48%. At N=1, mean unsupported numbers fall from 0.353 to 0.206 (41.6%). At N=8, full-system coverage is 84.82%, versus 82.55% for SFT. The finite-budget evidence therefore makes the adaptation's candidate-supply role visible alongside the final selection result.

The resulting insight is the complementary role of adaptation and verification: generator behavior determines whether a finite pool supplies a numerically participating, rule-satisfying candidate, while verification determines the delivered answer. We emphasize both quantities in the prepared revision.

Additional controlled comparison: strong APIs with matched candidate budgets

Following your recommendation, we generated eight candidates per question from two strong proprietary API endpoints, using the same prompts, materials, verifier, and stable selection rule on all 573 questions:

Generator + N=8 selectorAnswers with unmatched numbersInformative-clean coverageNumerical omission
IR-Agent (4B)0/57384.82%15.18%
GPT endpoint1/57334.21%65.62%
Gemini endpoint17/57391.80%5.41%

Using only 4B parameters, IR-Agent combines zero verifier-detected violations in the final selected answers with substantial numerical participation. Gemini provides greater coverage, while IR-Agent satisfies the numerical rule on every selected answer in this test; relative to GPT, IR-Agent provides substantially greater coverage. The prepared revision includes N=1/2/4/8 results (Section 4.4 and Appendix B).

Informative-clean coverage requires g>=1 and f=0 in the selected answer; omission means g=0. These are numerical-string measures. The new API endpoint runs control candidate counts and visible-output scoring length; API sampling defaults and internal reasoning compute differ from the local runs. Historical single-response API results are reported separately.

Q2. Semantic-level factuality evaluation and joint multi-objective optimization

Your questions helped sharpen the evidence boundary. Independent claim-level semantic assessment and a same-training-budget joint-reward comparison remain open. The new API experiment supplies the matched-candidate inference control; the revision defines decoupling as the allocation of objectives, without asserting demonstrated superiority over joint rewards. AI-assisted screening is preparatory evidence organization, not a completed human assessment.

For register, IR-Agent has 2.66% source four-gram overlap and 93.02% active-expression incidence at N=8. Following your comment, the prepared revision labels these as surface diagnostics and reports answer length alongside them, rather than equating them with human-rated naturalness or completeness.

Thank you again for helping improve the work. With approximately two days remaining in the discussion period, we would appreciate your feedback on whether these additions address your concerns and your consideration of the new evidence when updating your assessment.

Best regards, The authors

Reference: Ibrahim, Hafner & Rocher. “Training language models to be warm can reduce accuracy and increase sycophancy.” Nature (2026). https://doi.org/10.1038/s41586-026-10410-0

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.