- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
IR-Agent combines supervised fine-tuning, numerical reward training, and selection from eight candidate answers for company-side earnings-call responses. The paper also introduces IRCorpus and rule-based measures of numerical support, omission, and writing style.
Critical strengths
-
The paper studies a clear practical task and builds a corpus with separate companies for training, validation, and testing.
-
The three stages give a simple and understandable way to combine imitation, numerical training, and response selection.
Critical weaknesses
-
The verifier ignores the financial item, period, sign, and unit, and even rejects the correct value 937 when the source uses 937.00 in Appendix A.8. Without an independent audit, its scores do not establish how often the financial claims are actually correct.
-
Both SFT plus selection and full IR-Agent have zero detected fabrication on the test set. RLOO reduces test omission from 17.5% to 15.2% but raises validation omission from 16.9% to 20.8%, so its added value is not consistent across the two sets.
-
The API baselines provide one answer per question, while IR-Agent selects from eight candidates using the evaluation verifier. The paper also cannot confirm that the API models received the same instruction, so this comparison does not establish an advantage under matched conditions.
-
Every retained test reference contains numbers that pass the verifier, so the benchmark covers a selected group of questions. It does not establish how well the system handles missing information or decides when a reply without numbers is appropriate.
Questions for authors
-
Is the restriction to previously disclosed numbers a company policy or a benchmark assumption?
-
How do the strongest API baselines perform with the same prompt, eight candidates, and the same selection rule?
-
How does the decoupled design compare with a method that uses both style and numerical rewards under the same training budget?
5 comments
Thank you for the new experiments and for the revised tables. I have one question about the omission results.
Table 3 reports test omission of 17.5 percent for SFT plus selection and 15.2 percent for the full system. Table 6 reports 16.9 percent and 20.8 percent on validation. The direction is opposite in the two splits, and the two are reported apart.
If the two splits are pooled, SFT plus selection omits on 144 of 833 questions and the full system omits on 141 of 833. That is 17.3 percent against 16.9 percent, so the 2.3 point gain on test becomes 0.4 points once validation is counted.
Could you report this pooled comparison and say whether the difference remains meaningful? If you think the two splits should not be pooled, could you explain what makes them different enough to report separately?
Pooled omission comparison and clarification of the adaptation benefit
Dear Reviewer 6m1T,
Thank you for checking the two tables together and suggesting this useful summary. We have independently verified your calculation. At N=8, the split-specific and pooled omission counts are:
| Split | SFT + selection | Full system |
|---|---|---|
| Test | 100/573 (17.45%) | 87/573 (15.18%) |
| Validation | 44/260 (16.92%) | 54/260 (20.77%) |
| Pooled | 144/833 (17.29%) | 141/833 (16.93%) |
The pooled difference is 0.36 percentage points, or three questions. We agree that this small descriptive difference should not be interpreted as a meaningful overall omission advantage at N=8.
We originally reported the splits separately because validation was used for development and the companies are disjoint. This explains their different evaluation roles, but does not preclude a pooled descriptive summary. We will place the pooled comparison alongside both split-specific results, labeling it as a development-plus-test summary rather than an independent held-out estimate.
Your observation helps us clarify where disclosure shaping contributes most clearly: at limited candidate budgets. On the test set at N=2, under the same selector, it reduces all-dirty pools from 45 to 20 (55.6%) and increases informative-clean coverage from 45.72% to 51.48%. We will distinguish this finite-budget benefit from the similar aggregate omission of the two policies at N=8.
Thank you again for helping make the contribution and evaluation more precise.
Best regards, The authors
Response to Reviewer 6m1T: requested API control and numerical sensitivity results
Dear Reviewer 6m1T,
Thank you for your constructive questions and for recognizing the clear task formulation, company-disjoint corpus, and three-stage design. Your suggestions helped us strengthen the experiments. We have prepared a revised manuscript and answer your three questions below.
Q1. Is the restriction to previously disclosed numbers a company policy or a benchmark assumption?
It is the evidence-boundary assumption of this benchmark and assistant: the system answers from the supplied company materials, and the verifier measures adherence to that boundary. We have clarified this in the prepared revision. It is not a universal company policy or a legal-compliance test.
The retained references contain at least one source-matching number, allowing us to study numerical participation under this evidence constraint. This defines the numerical-answer subset being evaluated; cases where materials lack an answer require a separate abstention evaluation.
Q2. How do strong APIs perform with the same prompt, eight candidates, and selection rule?
Following your recommendation, we completed this comparison on all 573 common questions with identical system instructions, materials, eight-candidate budgets, and stable selection:
Using only 4B parameters, IR-Agent achieves zero verifier-detected violations in the final selected answers while retaining 84.82% informative-clean coverage. Your suggested control therefore provides stronger evidence for its usefulness as a compact, numerically constrained system. Gemini provides higher coverage, while IR-Agent satisfies the numerical rule on every selected answer in this test; GPT has low violations but substantially lower numerical coverage.
Informative-clean coverage requires g>=1 and f=0 in the selected answer; omission means g=0. The prepared revision includes the complete N=1/2/4/8 results (Section 4.4 and Appendix B) and the shared instruction (Appendix A.4). These new API endpoint runs are separate from the historical single-response results. Candidate budgets and visible-output scoring length are controlled, while API sampling distributions and internal reasoning compute differ from local runs.
Q3. How does decoupling compare with joint style and numerical rewards under the same training budget?
Our design assigns register supervision to demonstrations and numerical behavior to subsequent adaptation. This motivation is supported by Ibrahim et al., Nature (2026), who find that warmth fine-tuning can reduce factual accuracy and increase sycophancy. Our application concerns managerial register and evidence-bound numerical use; separating responsibilities avoids directly treating surface style proxies as a complete communication-quality objective.
The present controls evaluate the contribution of numerical adaptation within this design. At N=1, mean unsupported numbers decrease from 0.353 to 0.206 (41.6%). At N=2, all-dirty pools fall from 45 to 20 (55.6%) and informative-clean coverage rises from 45.72% to 51.48%. These provide finite-budget evidence for the adaptation stage. A same-training-budget joint-reward arm remains open; the revision presents objective allocation as the evaluated design without claiming established superiority over joint rewards.
Your 937 versus 937.00 example
Thank you for highlighting this distinction. It motivated a sensitivity check that holds the N=8 selected answers fixed and converts extracted strings to Decimal values. Unsupported-number totals remain unchanged for Base, SFT, IR-Agent, and GPT; Gemini changes from 18 to 17. Thus, the main comparison is maintained after numerical-value normalization. We report this separately in Appendix B. The verifier establishes numerical membership, while independent semantic assessment must also check metric, entity, unit, sign, and period associations.
Adaptation across splits
Your observation helped us distinguish finite-budget effects from the final N=8 endpoint. Both SFT and IR-Agent reach zero detected violations at N=8; the revision retains validation and test results and emphasizes where adaptation improves candidate supply, rather than presenting omission improvements as uniform across splits.
Thank you again for helping improve the work. With approximately two days remaining in the discussion period, we would appreciate your feedback on these additions and your consideration of the new evidence when updating your assessment.
Best regards, The authors
Reference: Ibrahim, Hafner & Rocher. Nature (2026). https://doi.org/10.1038/s41586-026-10410-0