- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces BhashaBench-Agentic, a multilingual (19 Indian languages), multi-turn benchmark for agentic tool use. Where the earlier BhashaBench-Multi asks whether a model knows a fact, this benchmark asks whether it can act on that content inside a conversation: ask when a detail is missing, confirm before changing state, call the right tool with correct arguments, and ground its answer in what the tool returned. The design centers on a six-action turn loop (clarify, intent, confirm, call, respond, halt), a ten-code trajectory taxonomy, and a four-signal scorer (execution, process, call-match, policy) combined by a strict rule where an episode counts as solved only if all four signals pass. The corpus has 4,258 episodes from 82 seed scenarios across four curation pipelines (hand-authored, mined from BhashaBench-Multi, persona-conditioned, de-identified Supreme Court judgments), with build-time validity gates, a cross-language parity test, and a ~55,000-case mutation suite to verify the scorer. Twelve models are reported, two in depth, and a worked case study traces a low score to a specific failure mode — the model recognizing a real case and responding as if a tool had confirmed it, producing fabricated citations — which a minimal disclosed prompt patch measurably improves.
Critical strengths
- A text multi-turn agentic tool-use benchmark across 19 Indian languages fills a real gap between static Indic benchmarks and English-only agent benchmarks.
- The paper clearly articulates what evidence it establishes (source ablation against contamination, scorer vs. construct validity, etc.)
- The provided case study strengthens the diagnostic value of the benchmark.
Critical weaknesses
- Over 4k episodes are constructed by merely 82 seeds. This confounds seed-wise evaluation vs cross-seed performance. Seeds corresponding to some areas might also be significantly underrepresented.
- The agentic structure is authored once and shared across languages. Only the surface dialogue is translated. This may or may not represent a real multilingual situation in practice.
- Though the paper points out that the validation suite validates the scorer, not realism, the applicability of the benchmark may be undermined due to insufficient external realism validity of multi-turn dynamics in different languages.
- Only two models are studied in depth, and the mandi domain reads a live price endpoint with no snapshot, so some scores are not reproducible.
Questions for authors
- Since the agentic structure is shared across languages, what does the benchmark reveal about language-specific agentic behavior beyond surface-dialogue robustness?
- How sensitive are domain-level conclusions, especially those with a small seed count?
2 comments
Response to Reviewer 5xTw (2/2)
Q2. Sensitivity of domain-level conclusions
We computed seed-clustered 95% intervals for every model in every domain, and the stability of each domain's model ranking under seed bootstrap (1,000 resamples). Domains with many seeds are tight: doc_qa (21 seeds) has a mean interval width of 0.09 and a median rank tau of 0.96; legal (10 seeds) 0.10 and 0.93; legal_rc (21 seeds) 0.11 and 0.93. Domains with few seeds are not: upi (4 seeds) has an interval width of 0.29, mandi (2 seeds) 0.30 and agri (2 seeds) 0.26, and their 5th-percentile rank tau is 0.62 to 0.80.
So domain-level conclusions for agri, mandi and upi are indicative only, and the revised paper says so and reports these intervals (§9, Table 22). The headline uses fixed domain weights, and its ranking is stable (W1).
Thank you again for the thoughtful review. We are happy to answer any further questions.
Response to Reviewer 5xTw (1/2)
We thank the reviewer for a careful review and for the focus on seeds as the unit of evidence, which sharpened how we report uncertainty.
W1. 4k episodes from only 82 seeds
We agree the seed, not the episode, is the unit of evidence, and all our statistics treat it so. Every interval in the paper and in this response is a seed-clustered bootstrap: seeds are resampled with all their episodes, so the intervals reflect 82 seeds, not 4,258 episodes.
Rankings are stable across seeds. Keeping a random half of the seeds in every domain (1,000 resamples), the median Kendall tau against the full 11-model ranking is 0.93 (5th percentile 0.86), and the two leading models, which are statistically tied, hold the top two places in 99% of resamples. Per-domain intervals are in our answer to Q2.
W2. Agentic structure shared across languages
This is deliberate. Holding the task fixed is what lets a difference between languages be attributed to language rather than to a different task, and it is what makes the finding in Q1 possible: across 19 languages models keep the interaction protocol but lose grounding.
The tasks are not language-neutral either. They are built on Indian settings (UPI payments, mandi prices, PM-KISAN and PMFBY eligibility, Supreme Court records), and some parts vary by language: romanised Hindi alongside Devanagari, code-mixed requests, answers and dates written in the language's own script and digits, and replies required in the episode's script. Language-specific task structure, such as a task that only arises in one language community, is outside the current design, and we state this scope explicitly.
W3. External realism of multi-turn dynamics
We agree that scripted dialogue is not real user behaviour, and we have no logs of real Indian-language users to compare against. What we can test is whether our conclusions depend on the scripted wording. We replaced the scripted user with an LLM-simulated user that says the same things in its own words and answers unexpected questions freely (CMHG W3): the headline score stays within run-to-run variation (0.312 against 0.271 to 0.324 for three scripted runs).
So the ranking does not hinge on the particular phrasing of the scripted turns. Whether real users in each language behave like our users, for example in how often they mix languages or leave out details, remains open, and we state it as a limitation.
W4. Two models in depth; mandi not reproducible
On depth: below is the per-signal breakdown for all 11 open models (corrected scorer, mean over 4,212 episodes). The four signals separate different failure modes: GLM-4.5 and Sarvam-30B fail mainly on call matching (0.36 and 0.40), Llama-4-Scout and the Sarvam models on execution (0.52 to 0.57), while process and policy compliance stay high for almost all models.
Model: execution / process / call match / policy / solved. gpt-oss-120b 0.83 / 0.92 / 0.86 / 0.98 / 0.41; Nemotron-3-Super 0.85 / 0.91 / 0.89 / 0.99 / 0.42; Gemma-4-26B 0.75 / 0.92 / 0.87 / 0.98 / 0.36; Qwen3-235B 0.84 / 0.92 / 0.86 / 0.99 / 0.36; DeepSeek-V4-Flash 0.80 / 0.91 / 0.73 / 0.97 / 0.22; MiniMax-M2.1 0.79 / 0.86 / 0.83 / 0.99 / 0.23; Mistral-Small-4 0.70 / 0.89 / 0.85 / 0.97 / 0.25; GLM-4.5 0.61 / 0.94 / 0.36 / 1.00 / 0.16; Llama-4-Scout 0.57 / 0.91 / 0.69 / 0.92 / 0.12; Sarvam-105B 0.52 / 0.78 / 0.75 / 0.91 / 0.06; Sarvam-30B 0.57 / 0.78 / 0.40 / 0.90 / 0.02.
On mandi: you are right, and we confirmed it. Replaying the same stored transcripts against the live price endpoint on different days moved mandi success between 0.50 and 0.10 for the same model. The prices each model actually saw are recorded in its transcript, so a snapshot built from them makes mandi scores replayable; the revised paper states this, and that mandi results are indicative only (§9).
Q1. What it shows about language-specific agentic behaviour
Because the task is held fixed, the benchmark can say which agentic behaviour degrades with language. Averaged over the 11 models (corrected scorer), the interaction protocol hardly moves across the 19 languages: asking for clarification when required is 0.78 to 0.82, confirming before a write is 0.98 to 1.00, and call economy is 0.65 to 0.70 in every language. What degrades is grounding: delivering the verified answer drops from 0.65 in English and 0.58 in Hindi to 0.46 in Konkani and Dogri and 0.32 in Bodo, and matching every gold call drops from 0.66 to 0.32. Strict success follows the same pattern, from 0.36 (English) to 0.11 (Bodo).
So in lower-resource languages models still follow the interaction protocol, but they less often issue the right tool call and carry its result into the reply. That is an agentic failure, not only a reading one. (For Bodo, part of the gap may come from translation quality; see CMHG W2c.) The revised paper adds this per-language, per-check table (Table 21, App. E.7).