acceptodds
Under review as a conference paper at ICLR 2027

BhashaBench-Agentic: A Multilingual, Multi-Turn Benchmark for Agentic Tool Use

Abstract

An assistant that can look up a government scheme in English proves little about whether it can be trusted to do the same in Hindi. Agentic assistants are already being deployed across Indian public-service and fintech settings, where people write in Devanagari, in romanized Hindi, or in sentences that switch between Hindi and English mid-thought and today's agentic benchmarks were built without that user in mind. We present BhashaBench-Agent to address that gap directly. Rather than another multiple-choice test of factual knowledge, it evaluates whether an agent can act correctly inside a multi-turn conversation: asking when a required detail is missing, confirming before any state-changing action, recovering from a failed tool call, and grounding its answer in what a tool actually returned rather than what sounds plausible. The corpus comprises 4,258 scored episodes built from 82 seed scenarios across 17 tools and 19 languages, spanning agricultural advisory, government scheme eligibility, legal and general-knowledge document work, mandi price lookup, and UPI payments, drawn from four distinct data-curation pipelines : hand-authored, persona-conditioned, mined from BhashaBench-Multi, and mined from de-identified Supreme Court judgments. Every pipeline shares one design choice that keeps the benchmark honest: a single gold trace per scenario, authored once and shared unchanged across every language variant, so a model cannot be handed an easier question in one language than another a property we verify with a build-time test rather than assert. Each episode is scored on four signals : execution, process, call-matching, and policy : combined as a product, so a fluent response in one turn cannot compensate for a safety failure in another. Before trusting any model's score, we validated the scorer itself against an 8-axis,  55,000-case mutation and golden-replay suite. We report results for twelve open-weight and frontier models, and use two of them for a detailed case study: tracing a low score to a specific, ground-truth-verified mechanism, a model declaring intent to call a tool, then answering as though it already had, producing a fabricated citation in the process and testing a minimal, disclosed prompt intervention against it. The corpus, harness, and scorer are released alongside this paper.

Primary area:
datasets and benchmarks
Keywords:
Agentic, Tool Calling, Benchmark, Datasets, Multilingual, LLM Evaluation, Multi-Turn Dialogue, Indic Languages

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5xtWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BhashaBench-Agentic, a multilingual (19 Indian languages), multi-turn benchmark for agentic tool use. Where the earlier BhashaBench-Multi asks whether a model knows a fact, this benchmark asks whether it can act on that content inside a conversation: ask when a detail is missing, confirm before changing state, call the right tool with correct arguments, and ground its answer in what the tool returned. The design centers on a six-action turn loop (clarify, intent, confirm, call, respond, halt), a ten-code trajectory taxonomy, and a four-signal scorer (execution, process, call-match, policy) combined by a strict rule where an episode counts as solved only if all four signals pass. The corpus has 4,258 episodes from 82 seed scenarios across four curation pipelines (hand-authored, mined from BhashaBench-Multi, persona-conditioned, de-identified Supreme Court judgments), with build-time validity gates, a cross-language parity test, and a ~55,000-case mutation suite to verify the scorer. Twelve models are reported, two in depth, and a worked case study traces a low score to a specific failure mode — the model recognizing a real case and responding as if a tool had confirmed it, producing fabricated citations — which a minimal disclosed prompt patch measurably improves.

Critical strengths
  1. A text multi-turn agentic tool-use benchmark across 19 Indian languages fills a real gap between static Indic benchmarks and English-only agent benchmarks.
  2. The paper clearly articulates what evidence it establishes (source ablation against contamination, scorer vs. construct validity, etc.)
  3. The provided case study strengthens the diagnostic value of the benchmark.
Critical weaknesses
  1. Over 4k episodes are constructed by merely 82 seeds. This confounds seed-wise evaluation vs cross-seed performance. Seeds corresponding to some areas might also be significantly underrepresented.
  2. The agentic structure is authored once and shared across languages. Only the surface dialogue is translated. This may or may not represent a real multilingual situation in practice.
  3. Though the paper points out that the validation suite validates the scorer, not realism, the applicability of the benchmark may be undermined due to insufficient external realism validity of multi-turn dynamics in different languages.
  4. Only two models are studied in depth, and the mandi domain reads a live price endpoint with no snapshot, so some scores are not reproducible.
Questions for authors
  1. Since the agentic structure is shared across languages, what does the benchmark reveal about language-specific agentic behavior beyond surface-dialogue robustness?
  2. How sensitive are domain-level conclusions, especially those with a small seed count?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 5xTw (1/2)

We thank the reviewer for a careful review and for the focus on seeds as the unit of evidence, which sharpened how we report uncertainty.

W1. 4k episodes from only 82 seeds

We agree the seed, not the episode, is the unit of evidence, and all our statistics treat it so. Every interval in the paper and in this response is a seed-clustered bootstrap: seeds are resampled with all their episodes, so the intervals reflect 82 seeds, not 4,258 episodes.

Rankings are stable across seeds. Keeping a random half of the seeds in every domain (1,000 resamples), the median Kendall tau against the full 11-model ranking is 0.93 (5th percentile 0.86), and the two leading models, which are statistically tied, hold the top two places in 99% of resamples. Per-domain intervals are in our answer to Q2.

W2. Agentic structure shared across languages

This is deliberate. Holding the task fixed is what lets a difference between languages be attributed to language rather than to a different task, and it is what makes the finding in Q1 possible: across 19 languages models keep the interaction protocol but lose grounding.

The tasks are not language-neutral either. They are built on Indian settings (UPI payments, mandi prices, PM-KISAN and PMFBY eligibility, Supreme Court records), and some parts vary by language: romanised Hindi alongside Devanagari, code-mixed requests, answers and dates written in the language's own script and digits, and replies required in the episode's script. Language-specific task structure, such as a task that only arises in one language community, is outside the current design, and we state this scope explicitly.

W3. External realism of multi-turn dynamics

We agree that scripted dialogue is not real user behaviour, and we have no logs of real Indian-language users to compare against. What we can test is whether our conclusions depend on the scripted wording. We replaced the scripted user with an LLM-simulated user that says the same things in its own words and answers unexpected questions freely (CMHG W3): the headline score stays within run-to-run variation (0.312 against 0.271 to 0.324 for three scripted runs).

So the ranking does not hinge on the particular phrasing of the scripted turns. Whether real users in each language behave like our users, for example in how often they mix languages or leave out details, remains open, and we state it as a limitation.

W4. Two models in depth; mandi not reproducible

On depth: below is the per-signal breakdown for all 11 open models (corrected scorer, mean over 4,212 episodes). The four signals separate different failure modes: GLM-4.5 and Sarvam-30B fail mainly on call matching (0.36 and 0.40), Llama-4-Scout and the Sarvam models on execution (0.52 to 0.57), while process and policy compliance stay high for almost all models.

Model: execution / process / call match / policy / solved. gpt-oss-120b 0.83 / 0.92 / 0.86 / 0.98 / 0.41; Nemotron-3-Super 0.85 / 0.91 / 0.89 / 0.99 / 0.42; Gemma-4-26B 0.75 / 0.92 / 0.87 / 0.98 / 0.36; Qwen3-235B 0.84 / 0.92 / 0.86 / 0.99 / 0.36; DeepSeek-V4-Flash 0.80 / 0.91 / 0.73 / 0.97 / 0.22; MiniMax-M2.1 0.79 / 0.86 / 0.83 / 0.99 / 0.23; Mistral-Small-4 0.70 / 0.89 / 0.85 / 0.97 / 0.25; GLM-4.5 0.61 / 0.94 / 0.36 / 1.00 / 0.16; Llama-4-Scout 0.57 / 0.91 / 0.69 / 0.92 / 0.12; Sarvam-105B 0.52 / 0.78 / 0.75 / 0.91 / 0.06; Sarvam-30B 0.57 / 0.78 / 0.40 / 0.90 / 0.02.

On mandi: you are right, and we confirmed it. Replaying the same stored transcripts against the live price endpoint on different days moved mandi success between 0.50 and 0.10 for the same model. The prices each model actually saw are recorded in its transcript, so a snapshot built from them makes mandi scores replayable; the revised paper states this, and that mandi results are indicative only (§9).

Q1. What it shows about language-specific agentic behaviour

Because the task is held fixed, the benchmark can say which agentic behaviour degrades with language. Averaged over the 11 models (corrected scorer), the interaction protocol hardly moves across the 19 languages: asking for clarification when required is 0.78 to 0.82, confirming before a write is 0.98 to 1.00, and call economy is 0.65 to 0.70 in every language. What degrades is grounding: delivering the verified answer drops from 0.65 in English and 0.58 in Hindi to 0.46 in Konkani and Dogri and 0.32 in Bodo, and matching every gold call drops from 0.66 to 0.32. Strict success follows the same pattern, from 0.36 (English) to 0.11 (Bodo).

So in lower-resource languages models still follow the interaction protocol, but they less often issue the right tool call and carry its result into the reply. That is an agentic failure, not only a reading one. (For Bodo, part of the gap may come from translation quality; see CMHG W2c.) The revised paper adds this per-language, per-check table (Table 21, App. E.7).

AuthorsOpenReview ↗

Response to Reviewer 5xTw (2/2)

Q2. Sensitivity of domain-level conclusions

We computed seed-clustered 95% intervals for every model in every domain, and the stability of each domain's model ranking under seed bootstrap (1,000 resamples). Domains with many seeds are tight: doc_qa (21 seeds) has a mean interval width of 0.09 and a median rank tau of 0.96; legal (10 seeds) 0.10 and 0.93; legal_rc (21 seeds) 0.11 and 0.93. Domains with few seeds are not: upi (4 seeds) has an interval width of 0.29, mandi (2 seeds) 0.30 and agri (2 seeds) 0.26, and their 5th-percentile rank tau is 0.62 to 0.80.

So domain-level conclusions for agri, mandi and upi are indicative only, and the revised paper says so and reports these intervals (§9, Table 22). The headline uses fixed domain weights, and its ranking is stable (W1).

Thank you again for the thoughtful review. We are happy to answer any further questions.

Reviewer_aWj5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BhashaBench-Agentic evaluates multilingual tool use through 4,258 episodes derived from 82 seeds, covering 19 languages and 17 tools. It scores execution, process, call matching, and policy compliance, requiring all applicable signals to pass. The paper reports twelve models under a common patched protocol and studies two models before and after prompt changes. The legal-task case study identifies fabricated references in responses produced without executing the required tool calls. The intended contribution is a multilingual, stateful benchmark with auditable scoring and failure diagnosis.

Critical strengths
  1. The benchmark addresses a useful gap in Indic-language evaluation: acting correctly across turns, including clarification, confirmation, and grounding, rather than only answering knowledge questions. Shared scenario state and reference traces are sensible foundations for comparisons across languages.

  2. The authors invest substantial effort in scorer inspection. Gold replay, targeted mutations, argument-reachability checks, and seed-clustered intervals are valuable safeguards. The case study checks fabricated identifiers against the actual scenario and harness, and the prompt intervention includes a 2×2 experiment. This is more informative than an unexplained leaderboard score.

Critical weaknesses

The central cross-language validity claim is not yet established. Sharing a gold trace prevents reference re-authoring, but does not ensure equivalent dialogue meaning or equal scoring error across languages. Appendix D.15 reports machine-translated dialogue for 13 seeds without completed native-speaker review; script conformity cannot validate meaning. Appendix B.3 uses keyword overlap for answers, and D.11 uses token-overlap F1 for query matching despite §4.4 describing meaning-based matching. These checks can reject valid paraphrases unevenly. A bilingual audit of matched episodes and scorer false positives/negatives is needed before attributing language gaps to model ability.

Questions for authors

On a manageable sample of paired language variants, how often do bilingual reviewers find equivalent task meaning, and how often do the answer/query scorers disagree with human correctness judgments? If an English search query retrieves the correct evidence in a non-English episode, which explicit task requirement makes it incorrect?

1 comment
AuthorsOpenReview ↗

Response to Reviewer aWj5

We thank the reviewer for a careful review. The question on matching led us to find and fix real scorer issues, which made the benchmark more accurate.

W1. Cross-language validity not established

We agree that a shared gold does not by itself guarantee equal meaning. For the rebuttal we measured it with a round-trip check: about 70 distinct dialogue turns per language were translated back into English by a different model and rated for whether the original meaning survived. In the median language 79% of turns keep their meaning, against 96% for the hand-corrected Hindi. Bodo is the clear exception at 22%, and we mark its results as lower-confidence (details in our answer to CMHG W2c).

On scoring error: following your W2, we found and fixed real cross-language scoring errors (answers in the episode's own script, search queries in another language, dates in native digits) and rescored every transcript.

After these fixes, the per-language breakdown over the 11 models shows that the interaction checks are nearly flat across languages: clarify-when-required is 0.78 to 0.82 and confirm-before-write 0.98 to 1.00. The gap sits in grounding: the verified answer is delivered in 0.65 of English episodes and 0.32 of Bodo ones, and every gold call is matched in 0.66 and 0.32. Bodo is lowest on both grounding and translation quality, so part of its gap may come from translation; in the other languages the gap appears in grounding but not in the interaction checks, which depend on understanding the same dialogue.

A native-speaker review in every language is the natural next step, and we note it in the limitations.

W2. Keyword and token-F1 matching may reject valid paraphrases

Thank you; this concern was right, and following it up found concrete errors. We fixed four issues and re-scored every stored transcript.

(1) kb answers in the episode's own script. The knowledge base stores native-script answers for only a few languages; elsewhere the reply had to contain the English string, so a correct reply in the episode's script (মোনোসাইট for Monocyte) was scored wrong. The reply is now accepted if it names the verified answer in any Indic script (script-normalised consonant match against the Hindi form; numbers matched exactly). kb success rises sharply (Nemotron 0.43 to 0.72, Qwen3 0.32 to 0.67).

(2) Search queries in another language or wording. The query argument had to match the user's own question; it now also passes if it matches the same question in any language version of the episode (Q1).

(3) Translated payment requests. In 46 of 200 UPI episodes, in 8 languages, machine translation dropped the amount (the user asks to send '0' rupees), so no agent could succeed. We exclude these episodes from all reported numbers.

(4) Dates in Indian-script digits. A mandi price quoted with its date written in native digits was scored as undated; the check now normalises digits.

All checks stay deterministic and gold replay still scores 1.000. With all fixes the scores rise by 0.015 to 0.087 and the ranking is essentially unchanged (Kendall tau 0.96 against the submission): gpt-oss-120b 0.399, Nemotron-3-Super 0.389, Gemma-4-26B 0.318, Qwen3-235B 0.270, DeepSeek-V4-Flash 0.222, MiniMax-M2.1 0.191, Mistral-Small-4 0.185, GLM-4.5 0.118, Llama-4-Scout 0.107, Sarvam-105B 0.046, Sarvam-30B 0.042. The only change at the top is that gpt-oss-120b and Nemotron-3-Super swap first and second; they are statistically tied (seed-clustered 95% interval for the difference [-0.06, 0.08]). The revised paper reports the corrected leaderboard and corrects §4.4's wording.

Q1. English search query in a non-English episode

None; you are right. The free-text query argument is compared, by token F1, only with the user's own question in the episode's language, so a query in English, or in another Indian language, that retrieves the same evidence fails call-match. Because the task asks the agent to find the evidence, not to search in a particular language, we changed the scorer: the query also passes if it matches the same question in any of the 19 language versions of the episode (same seed and trajectory).

On its own this raises scores by 0.000 to 0.060; the change is in kb, kisan and agri. gpt-oss-120b, which often issues English queries, is affected most (0.311 to 0.372), then MiniMax-M2.1 (+0.029); the rest move by at most 0.015. Together with the other fixes (W2) the corrected leaderboard is: gpt-oss-120b 0.399, Nemotron-3-Super 0.389, Gemma-4-26B 0.318, Qwen3-235B 0.270, DeepSeek-V4-Flash 0.222, MiniMax-M2.1 0.191, Mistral-Small-4 0.185, GLM-4.5 0.118, Llama-4-Scout 0.107, Sarvam-105B 0.046, Sarvam-30B 0.042. gpt-oss and Nemotron-3-Super swap first and second but are statistically tied (95% interval for the difference [-0.06, 0.08]). The revised paper reports the corrected leaderboard and notes the tie (§7).

Thank you again for the thoughtful review. We are happy to answer any further questions.

Reviewer_CMHGOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces BhashaBench-Agentic, a multi-turn, tool-using benchmark in 19 Indian languages built from BhashaBench-Multi's domain content (agriculture, finance, law, Ayurveda). Each episode combines three things:

  • a seed scenario (user goal, sandbox state, tools);
  • a language/script pair;
  • a trajectory code from a ten-code taxonomy (eight are populated).
  • One gold trace per seed is shared across all languages. Free-text gold arguments are stored as pointers to user turns. Episodes are scored on four signals: execution, process, call-match and policy. An episode counts as solved only if every applicable signal passes.
Critical strengths
  1. A needed benchmark, with a design that makes cross-language comparison valid. No existing benchmark tests multi-turn, stateful tool use across this many Indic languages, including low-resource ones such as Bodo, Dogri and Maithili. The key design idea is the language-invariant gold trace, with turn pointers for free-text arguments, enforced by a build-time test. It keeps each language from getting its own reference answer, so per-language score differences are not caused by how the gold was written.
Critical weaknesses
  1. The evaluation does not establish what the score measures or how models rank.
  • No frontier models. All twelve models are self-hosted open-weight models (App. E). With a best score of 0.332, it is unclear whether the benchmark is hard or the tested models are mid-tier.
  • The leaderboard depends on a fitted prompt. Ten of the twelve models were run only with two prompt patches. The patches were designed around the failures of gpt-oss-120b and Qwen3-235B-A22B, and the authors say Patch 1 is "close to describing the harness's own scoring mechanics."
  • Tool use is not shown to be necessary. There is no no-tool baseline. In the kb domain (532 episodes of public exam facts), the scorer requires the tool call, but the task's information does not.
  • Task diversity is limited, and the paper does not justify it. The 4,258 episodes are 82 seeds multiplied by language, content-fill and templated trajectory variants. They use 17 tools in only 6 tool-set combinations (§B.5). 48 of the 82 seeds are retrieval or reading comprehension, and all write tasks come from 4 UPI seeds. §B.5's rationale, keeping trajectory shapes comparable, explains the fixed templates. It does not explain the small number of distinct tasks. No external validation. Scores are not compared with human judgment or with another agentic benchmark. Why it matters: the paper's core claim is that its score measures agentic ability. These gaps leave that claim largely untested.
  1. Gold correctness and the reported numbers cannot be trusted as written.
  • The legal gold comes from an LLM, with no audit. The gold for 1,102 Supreme Court episodes is derived from ledger fields that an LLM extracted, and no human audit is reported. §B.4 says a model's judgment "enters only the ledger-extraction stage, not the answer." That understates the problem, because the gold is computed from exactly those fields.
  • No expert review or inter-annotator checking is reported.
  • Translations are not checked for meaning. Hand-authored and persona dialogue (760 episodes) received only a script-conformance check. The paper does not describe how the Supreme Court dialogue was translated. Turn pointers carry the user's words into the gold, so a mistranslation becomes part of the gold. This confounds the low-resource results in Table 15.
  • The numbers are inconsistent. §8.3 reverses both of its comparisons; for example, it calls 24.3% "more" than 32.0%. l. 1299–1300 says all numbers reflect the kb_search correction, but §C.2 says Tables 13–14 predate it. The abstract pairs a mean computed before the correction (0.895) with the strict rate computed after it (0.215). The Supreme Court seed and episode counts disagree across Table 4, Table 8 and §B.4.
  1. Scripted user is used without strong enough rationale, and feels like a big weakening in "agency" aspect of the dataset. The argument would be strong if the author had had shown they tried simulated user, but find it working properly and use scripted user as an fallback.
Questions for authors
  1. Please reconcile the numbers:
  • which tables and values were computed before vs. after the kb_search correction (§D.11);
  • §D.5's call-match values vs. Table 6;
  • the Supreme Court seed and episode counts (Table 4 vs. Table 8 vs. §B.4);
  • the 21-of-32 relabelling vs. Table 4's 32 doc_qa seeds.
  1. Was any human or expert audit done on the LLM-extracted ledgers' established/disputed labels, or on the kisan eligibility rules? If so, what were the error rates?

  2. How was the Supreme Court dialogue translated and checked?

  3. How does an admitted BhashaBench-Multi row become an episode?

  4. Was the 57-episode development slice drawn from the evaluation corpus, and is it included in the full-corpus numbers?

  5. Is the small number of distinct seeds intentional, and if so, what justifies it beyond keeping trajectory shapes comparable? Please report:

  • the number of distinct seeds behind each trajectory code and interaction axis (Table 3);
  • the number of seeds that involve a write action;
  • whether model rankings stay stable under seed subsampling, e.g. rank agreement when resampling half the seeds.
  1. Why does the benchmark use a scripted user rather than an LLM-simulated one, as in τ-bench and τ²-bench? Please state the rationale and discuss the costs:
  • the user never challenges incorrect or fabricated answers (App. A);
  • escalation lines can supply the answer (turn [12] names D1);
  • reasonable but unanticipated clarifications could be penalised through the process signal.
  1. What metrics is used to compare model prediction and references? Token-overlap F1 is used for free-text tool arguments inside call-match. What about other comparisons?
5 comments
AuthorsOpenReview ↗

Response to Reviewer CMHG (1/5)

We thank the reviewer for a detailed and thoughtful review. Its suggestions on frontier models, validation and user simulation led to several of the most useful new experiments in this response. We answer each point below, across five comments.

W1a. No frontier models

We ran two frontier models for the rebuttal, Claude Sonnet 4.6 (closed) and Kimi K3 (open-weight), with the same harness, prompt and scorer, on a stratified subset of 240 episodes (30 per domain, fixed seed). On the same episodes, Claude Sonnet 4.6 scores 0.40 and Kimi K3 0.39, against 0.44 for gpt-oss-120b and 0.38 for Nemotron-3-Super, the best open models in the paper. Neither frontier model is separable from them (paired seed-bootstrap 95% intervals, e.g. Sonnet minus Nemotron [-0.07, 0.12]). Part of Kimi's run hit an API spending limit; on the 191 episodes it completed without errors it scores 0.46, still far from saturating the benchmark.

So the low ceiling is a property of the benchmark, not of mid-tier models: the strongest models we could run, closed and open, reach about 0.4 to 0.46. We read a sample of the frontier models' failed episodes. Some failures exposed scorer and data issues, which we fixed; all numbers here use the corrected scorer (see the general response). The remaining failures are of two kinds: behaviours the benchmark targets (answering a factual sub-question from memory instead of from the knowledge-base result, declaring a question unrelated to the case without opening the record, skipping a clarification the request needs), and breaches of operating rules stated in the system prompt (quoting a different market's price, executing a payment without first declaring the plan). The revised paper reports both frontier models (§7.1, App. E.1).

W1b. Leaderboard depends on a fitted prompt

The paper reports both patched models with and without the patches, and the patches are disclosed in full (App. D). Under the corrected scorer (general response) they raise gpt-oss-120b from 0.299 to 0.399 (+0.100) and Qwen3-235B from 0.118 to 0.270 (+0.152).

To check whether the patches also shape the other models' scores, we ran four models the patches were not designed around with and without them, over the full benchmark and on the same server for both conditions. The patches add +0.011 for Gemma-4-26B (seed-clustered 95% CI [-0.012, +0.040]), +0.030 for Llama-4-Scout ([+0.003, +0.050]), +0.067 for DeepSeek-V4-Flash ([+0.041, +0.087]) and +0.042 for MiniMax-M2.1 ([+0.023, +0.067]). For these models they mostly remove output-format failures, for example undecodable replies for DeepSeek (750 to 58 episodes) and whitespace loops for Llama (419 to 206).

The four models the patches were not written for keep exactly the same order with and without them (Gemma-4-26B, DeepSeek-V4-Flash, MiniMax-M2.1, Llama-4-Scout). Across all six models the ranking changes only where a target model moves up: gpt-oss-120b passes Gemma-4-26B for first place, and Qwen3-235B passes MiniMax-M2.1 for fourth. So the patches raise scores, mostly by fixing format failures, but they do not decide the ranking among the models they were not written for. The revised paper reports this comparison in §7.1 and Table 19.

W1c. Tool use is not shown to be necessary (kb domain)

We ran this baseline. gpt-oss-120b answered all 532 kb episodes with no tools available, and we compared it with the same model, prompt and server with tools. On the 456 episodes whose answer is in the knowledge base, the final reply contains the verified answer in 42% of episodes without tools and 78% with tools (English and Hindi: 67% vs 98%). Without tools the model answers confidently and wrongly; for example it names the neutrophil as the largest leukocyte.

So most of these exam facts cannot be answered reliably from memory, and the lookup is what makes the answer correct. We agree a share is answerable from memory; for those episodes kb tests whether the agent checks before answering, which is the grounding behaviour the domain is meant to measure. Excluding kb altogether leaves the ranking essentially unchanged (Kendall tau 0.96; only the two Sarvam models swap last and second-last). The revised paper adds this baseline and the kb-excluded result (§6).

AuthorsOpenReview ↗

Response to Reviewer CMHG (2/5)

W1d. Limited task diversity: 82 seeds, 17 tools, 4 write seeds

The seed count is deliberate: each seed is expanded across 19 languages and up to 8 trajectory codes with one shared gold, so a language gap cannot come from different tasks. The cost is fewer distinct tasks, and we agree write actions are thin (4 seeds, all UPI). We state this as a limitation.

Seed counts (a seed usually has several trajectories and axes, so the counts overlap): by trajectory, T0 81, T3 53, T6 57, T4 22, T8 5, T7 2, T5 1, T9 1; by interaction axis, grounded-answer 78, clarify 53, code-mixed 15, confirm-gate 4; write seeds 4 (all UPI). Trajectories with fewer than five seeds (T5, T7, T9) support no per-trajectory conclusion, and the revised paper says so (§9, App. E.8).

Rankings are stable under seed subsampling: keeping a random half of the seeds in every domain (1,000 resamples), the median Kendall tau against the full 11-model ranking is 0.93 (5th percentile 0.86). The two leading models are statistically tied (gpt-oss-120b minus Nemotron-3-Super, seed-clustered 95% interval [-0.06, 0.08]); they hold the top two places in 99% of resamples, and which of them is first varies.

W1e. No external validation

We agree this matters, and we cannot yet validate against an external measure. Part of the reason is that the reference does not exist: to our knowledge no text-based agentic benchmark covers Indian languages. The multilingual extensions of tau-bench cover other languages (τ-bench-LILT: German, Korean, Turkish, Farsi; SEA-τ: five Southeast Asian languages), and VoiceAgentBench tests spoken queries in a few Indic languages, not text tool use; the revised paper discusses all three in the related work (§3). Filling this gap is the purpose of the benchmark. Only five of our models have published English tau2-bench scores, mostly for different versions or reasoning settings, so a rank correlation on them would not be meaningful.

What we can show is that the score tracks agentic behaviour rather than language skill or memory. (1) Removing the tools collapses the answers: on kb facts, correct answers fall from 78% with tools to 42% without (W1c). (2) The score separates agentic behaviours: across the 19 languages, clarifying and confirming stay flat (0.78 to 0.82 and 0.98 to 1.00) while grounding the answer in a tool result falls from 0.65 to 0.32 (aWj5 W1); a test of language skill alone would not split this way. (3) Frontier models score highest or level with the best open models, and the smallest models lowest (W1a). (4) The failed episodes we inspected are agentic failures: answering without the lookup, declaring a question unrelated without opening the record, skipping a needed clarification. That inspection also found scorer issues, which we fixed (general response).

A human validation of the scorer, with two people judging a stratified sample of scored episodes, is the natural next step, and the revised paper notes it in the limitations (§9).

W2a. Legal gold comes from LLM-extracted ledgers, no audit

We checked a random sample of 50 ledger statements. The sample was drawn with a fixed seed, stratified by label in proportion to the corpus (38 established and 12 disputed, against 967 and 301 in the full ledgers), and spread round-robin over all 29 judgments, so that every judgment is represented.

Two members of our team checked every statement against the judgment independently, without seeing each other's answers. Each started from an LLM-drafted verdict and rationale citing the relevant paragraphs, and confirmed or corrected it. They are members of our team, not legal experts.

Each judged the content fully correct for 88% of statements (both for 82%); the rest were mostly imprecise compressions (e.g. a contractual appointment clause stated as an appointment), and one checker judged one statement wrong. The established/disputed label was judged correct for 94% and 84% of statements; both flagged the same 2 of 50 labels (4%), and at least one flagged 9 (18%). They agreed on 88% of content and 86% of label judgments (Cohen's kappa 0.44 and 0.30; kappa is low because almost all items are judged correct, so the few disagreements weigh heavily). Most label disagreements concern one definition: our ledger rule marks as disputed anything asserted by one side, even when the Court later resolves it.

We also traced where the sampled statements reach the benchmark: 10 of the 50 appear in scored episodes, 9 as the text of a record document and 1 as a claim the user asks the agent to check. None of the statements flagged with a label error, and not the one judged wrong, is used as such a claim.

The revised paper states the label definition explicitly (§5, App. E.5).

AuthorsOpenReview ↗

Response to Reviewer CMHG (3/5)

W2b. No expert review or inter-annotator agreement

Thank you. For the legal ledgers, two members of our team independently checked a random sample of 50 statements against the judgments; they agreed on 88% of content and 86% of label judgments (W2a).

The policy rules are the benchmark's own specification, as with tau-bench's policy documents: the agent reads the rule text in every episode and is scored against that same text, so the rules need to be consistent rather than match a real regulation, and the gold trajectory of every episode satisfies all of them.

The scheme facts that scored kisan episodes rely on (PM-KISAN exclusions, PMFBY premium shares) match the official scheme guidelines.

Expert review of the rules and native-speaker review of the translations, with agreement on a doubly-checked subset, are the natural next steps; we note their absence in the limitations.

W2c. Translations not checked for meaning

All dialogue, including the hand-authored, persona and Supreme Court episodes, was written in English and machine-translated into the other languages with an in-house Indic translation LLM at temperature 0. Outputs are cached in the repository, so every rebuild produces identical episodes. Hindi was then corrected by hand over several rounds. Before translating, we probed the model by hand in all 22 candidate languages and excluded four (sd, sat, ks, mni) whose output was unusable. Each language pack records this provenance.

Beyond script conformance, every translated turn passes automatic defect checks, and failing turns were retranslated: empty or truncated output, repeated words or phrases, character floods, invisible-character padding, leaked preambles such as the language's own name, and untranslated English. A separate gate then checks that every episode is still scoreable: the answer keywords survive translation and the reply is in the right script.

These checks catch broken output, not wrong meaning, so for the rebuttal we also ran a round-trip check. We sampled about 70 distinct dialogue turns per language, translated each back into English with a different model (Gemma-4-31B), and had it rate whether the meaning of the original English survived (1 to 5). In the median machine-translated language, 79% of turns keep their meaning (rated 4 or 5), against 96% for the hand-corrected Hindi. With about 70 turns per language each figure is uncertain by roughly ±10 points, so most of the spread between languages is sampling noise; only Dogri (65%) sits clearly lower. Bodo is the outlier at 22%, partly because the back-translating model is itself weak in Bodo, and we mark Bodo results as lower-confidence. A round trip also counts errors made on the way back, so these figures understate the forward translation.

Most gold answers do not depend on this translation: in the tool-grounded and legal domains they are tool results, record identifiers or ledger facts, which are language-independent, and in general reading comprehension they come from the source dataset's human translations. The exception is legal reading comprehension, where the answer keywords are translated with the passage and kept identical to it, so the agent can always find them. A native-speaker meaning check is the natural next step, and we note it in the limitations.

W2d. Reported numbers are inconsistent

Thank you for the careful reading; these are errors. We recomputed every figure from the stored transcripts, and the full list of corrections is in our answer to Q1. In short:

§8.3 had both comparisons reversed. The abstract's 0.895 was the pre-fix mean of the process signal alone; the four-signal mean after the fix is 0.870. The Supreme Court portion is 1,102 episodes from 20 seeds (10 legal_m seeds with 532 episodes, of which §B.4 wrongly says 9, and 10 lrc seeds with 570). Table 4 predates the relabelling of the reading-comprehension seeds: legal_rc and doc_qa have 21 seeds each, not 10 and 32.

Every table in the revised paper now states which scorer it uses and whether it is before or after the kb_search fix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We sincerely thank all three reviewers for their careful reading and constructive feedback. Their questions were specific and well judged, and working through them has made the paper clearly stronger. Below is a summary of what we did in response.

New experiments. Following the reviewers' suggestions, we ran:

  • two frontier models (Claude Sonnet 4.6 and Kimi K3) on a stratified 240-episode subset;
  • a no-tools baseline for kb;
  • an LLM-simulated user in place of the scripted one;
  • four more models, each with and without the prompt patches;
  • a round-trip meaning check of the translations in every language;
  • a check of 50 legal ledger statements by two members of our team.

We also added per-domain and per-language analyses with seed-clustered intervals.

Scorer fixes. The reviewers' questions led directly to an important improvement. While inspecting the frontier models' failed episodes, and following up aWj5's question on matching, we found and fixed four scorer and data issues:

  1. kb answers given in the episode's own script were scored as wrong;
  2. a search query in another language or wording failed call-match;
  3. in 46 of 200 UPI episodes, machine translation dropped the payment amount, so we now exclude these;
  4. mandi dates written in Indian-script digits were scored as missing.

All checks are still deterministic, and gold replay still scores 1.000. We are grateful for this question in particular, because it made the benchmark more accurate.

Corrected results. With the fixes, every model's score rises by 0.015 to 0.087, and the ranking is essentially unchanged (Kendall τ 0.96 against the submission). The corrected leaderboard is:

ModelScore
gpt-oss-120b0.399
Nemotron-3-Super0.389
Gemma-4-26B0.318
Qwen3-235B0.270
DeepSeek-V4-Flash0.222
MiniMax-M2.10.191
Mistral-Small-40.185
GLM-4.50.118
Llama-4-Scout0.107
Sarvam-105B0.046
Sarvam-30B0.042

The top two are statistically tied. The frontier models score about 0.40 on the subset, level with the best open models, so the benchmark is far from saturated. All numbers in our replies use the corrected scorer.

Revised paper. We have uploaded a revised paper, with changed text in blue. The main additions are:

  • §7.1: frontier models and the patch effect;
  • new paragraphs in §5, §6 and §9: translation, the ledger check, tool necessity, the simulated user, and limitations;
  • Appendix E: the details behind each result in this response.

We thank the reviewers again for their time and thoughtful engagement, and we welcome any further questions or suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.